Tables of results (Token analysis)

Total

All tokens

	War	Other	Ground Truth
4grams	4949	12751	69883
3grams	7613	19702	108922
2grams	13747	35235	196585

All tokens (by type) (numbers)

	War	Other	Ground Truth
4grams (AC)	133	191	2744
4grams (I)	481	1392	6854
4grams (L)	4335	11168	60285
3grams (AC)	195	286	4153
3grams (I)	528	1523	7512
3grams (L)	6890	17893	97257
2grams (AC)	226	489	7233
2grams (I)	641	1831	9094
2grams (L)	12770	32915	180258

All tokens (by type) (percentages)

	War	Other	Ground Truth
4grams (AC)	2%	1%	4%
4grams (I)	10%	11%	10%
4grams (L)	88%	88%	86%
3grams (AC)	2%	1%	4%
3grams (I)	7%	8%	7%
3grams (L)	91%	91%	89%
2grams (AC)	2%	1%	4%
2grams (I)	5%	5%	5%
2grams (L)	93%	93%	92%

Set Other

Token from set Other (All Caps)

	2grams	3grams	4grams
1	57	121	99
2 to 5	63	34	17
6 to 10	14	7	4
11 to 50	9	2	2
Total	143	164	122

Token from set Other (Initials)

	2grams	3grams	4grams
1	34	158	218
2 to 5	40	150	160
6 to 10	39	38	38
11 to 50	44	33	25
51 to 100	5	ø	ø
More than 100	1	ø	ø
Total	163	379	441

Token from set Other (Lowercases)

	2grams	3grams	4grams
1	61	390	1005
2 to 5	75	543	924
6 to 10	30	238	243
11 to 50	113	332	214
51 to 100	42	46	15
101 to 500	83	20	4
501 to 1000	10	ø	ø
More than 1000	2	ø	ø
Total	416	1569	2405

Tokens from set Other (11 and more)

	2grams	3grams	4grams
11 and more	309	433	260
Total	722	2112	2968
Percentage on the total	43%	21%	9%

Tokens from set Other (Only 1)

	2grams	3grams	4grams
1	152	669	1322
Total	722	2112	2846
Percentage on the total	21%	32%	46%

Tokens from set Other (Stopwords)

	2grams	3grams	4grams
Stopwords	44	27	32
Total	309	433	260
Percentage on the total	14%	6%	12%

Set War

Token from set War (All Caps)

	2grams	3grams	4grams
1	48	68	58
2 to 5	40	22	13
6 to 10	11	4	4
11 to 50	6	2	1
Total	105	96	76

Token from set War (Initials)

	2grams	3grams	4grams
1	32	101	119
2 to 5	43	61	58
6 to 10	10	12	9
11 to 50	20	10	8
Total	105	184	194

Token from set War (Lowercases)

	2grams	3grams	4grams
1	69	418	818
2 to 5	68	425	534
6 to 10	42	166	97
11 to 50	110	134	58
51 to 100	40	11	1
101 to 500	36	4
Total	365	1158	1508

Tokens from set War (11 and more)

	2grams	3grams	4grams
11 and more	211	161	68
Total	575	1438	1778
Percentage on the total	37%	11%	4%

Tokens from set War (Only 1)

	2grams	3grams	4grams
1	149	587	995
Total	575	1438	1778
Percentage on the total	26%	41%	56%

Tokens from set War (Stopwords)

	2grams	3grams	4grams
Stopwords	33	25	20
Total	211	161	68
Percentage on the total	16%	16%	29%

Set Ground Truth

Token from set Ground Truth (All Caps)

	2grams	3grams	4grams
1	27	365	528
2 to 5	79	359	303
6 to 10	34	81	34
11 to 50	101	53	29
51 to 100	25	9	3
More than 100	15	3	2
Total	281	870	899

Token from set Ground Truth (Initials)

	2grams	3grams	4grams
1	29	287	515
2 to 5	54	273	356
6 to 10	24	99	95
11 to 50	68	107	96
51 to 100	27	16	13
More than 100	22	13	10
Total	224	795	1085

Token from set Ground Truth (Lowercases)

	2grams	3grams	4grams
1	76	440	1349
2 to 5	88	531	1510
6 to 10	48	311	602
11 to 50	89	650	907
51 to 100	33	207	149
101 to 500	124	203	79
501 to 1000	44	17	8
More than 1000	52	7
Total	554	2366	4604

Tokens from set Ground Truth (11 and more)

	2grams	3grams	4grams
11 and more	600	1285	1296
Total	1059	4031	6588
Percentage on the total	57%	32%	20%

Tokens from set Ground Truth (Only 1)

	2grams	3grams	4grams
1	132	1092	2392
Total	1059	4031	6588
Percentage on the total	12%	27%	36%

Tokens from set Ground Truth (Stopwords)

	2grams	3grams	4grams
Stopwords	75	41	45
Total	600	1285	1296
Percentage on the total	13%	3%	3%

Comparison

Most popular tokens (11 and more) (numbers)

	War	Other	Common
2grams	0	98	211
3grams	5	277	156
4grams	7	199	61

	War	GT	Common
2grams	0	389	211
3grams	0	1124	161
4grams	0	1228	68

	Other	GT	Common
2grams	2	293	307
3grams	7	859	426
4grams	10	1046	250

	War_unique	GT	Common
2grams	0	600	0
3grams	0	1280	5
4grams	0	1289	7

	Other_unique	GT	Common
2grams	2	504	96
3grams	7	1015	270
4grams	10	1107	189

	Common_unique	GT	Common
2grams	0	389	211
3grams	0	1129	156
4grams	0	1235	61

Most popular tokens (11 and more) (percentages)

	War	Other
2grams	0%	32%
3grams	3%	64%
4grams	10%	77%

	War	GT
2grams	0%	64%
3grams	0%	88%
4grams	0%	95%

	Other	GT
2grams	0,6%	49%
3grams	1,6%	67%
4grams	3,8%	81%

	War_unique	GT
2grams	0%	100%
3grams	0%	99,6%
4grams	0%	99,5%

	Other_unique	GT
2grams	2%	84%
3grams	2,5%	79%
4grams	5%	85%

	Common_unique	GT
2grams	0%	65%
3grams	0%	88%
4grams	0%	95%

Least popular tokens (Only 1 occurrence) (numbers)

	War	Other	Common
2grams	120	123	29
3grams	475	557	112
4grams	801	1128	194

	War	GT	Common
2grams	139	122	10
3grams	543	1048	44
4grams	910	2307	85

	Other	GT	Common
2grams	144	124	8
3grams	589	1012	80
4grams	1122	2192	200

	War_unique	GT	Common
2grams	112	124	8
3grams	435	1052	40
4grams	730	2321	71

	Other_unique	GT	Common
2grams	117	126	6
3grams	481	1016	76
4grams	942	2206	186

	Common_unique	GT	Common
2grams	27	130	2
3grams	108	1088	4
4grams	180	2378	14

Least popular tokens (Only 1 occurrence) (percentages)

	War	Other
2grams	81%	81%
3grams	81%	83%
4grams	81%	85%

	War	GT
2grams	93%	92%
3grams	93%	96%
4grams	91%	96%

	Other	GT
2grams	95%	94%
3grams	88%	92%
4grams	85%	92%

	War_unique	GT
2grams	93%	94%
3grams	92%	96%
4grams	91%	97%

	Other_unique	GT
2grams	95%	95%
3grams	86%	93%
4grams	84%	92%

	Common_unique	GT
2grams	93%	98%
3grams	96%	99,6%
4grams	98%	99%