Opened .\URL-input-1M.txt with size 66,152,005
[ 2] 500 Q 972,542 E 27,462 H 4,966 D 3,867 I 2,431 R 344 C 318 L 2K
*** crawling 159.0 pps @ 15.2 Mbps
[ 4] 500 Q 958,217 E 41,787 H 6,999 D 5,495 I 3,522 R 502 C 474 L 7K
*** crawling 77.4 pps @ 13.7 Mbps
[ 6] 500 Q 942,234 E 57,770 H 9,193 D 7,293 I 4,707 R 688 C 663 L 11K
*** crawling 94.5 pps @ 11.3 Mbps
[ 8] 500 Q 931,152 E 68,852 H 10,827 D 8,642 I 5,656 R 839 C 824 L 16K
*** crawling 80.5 pps @ 10.1 Mbps
[ 10] 500 Q 919,806 E 80,198 H 12,425 D 9,945 I 6,519 R 952 C 933 L 17K
*** crawling 54.1 pps @ 5.7 Mbps
[ 12] 500 Q 907,551 E 92,453 H 14,012 D 11,264 I 7,415 R 1,067 C 1,049 L 20K
*** crawling 57.5 pps @ 7.6 Mbps
[ 14] 500 Q 896,911 E 103,093 H 15,467 D 12,464 I 8,243 R 1,189 C 1,165 L 22K
*** crawling 58.0 pps @ 11.7 Mbps
[ 16] 500 Q 887,410 E 112,594 H 16,816 D 13,611 I 9,043 R 1,303 C 1,286 L 25K
*** crawling 60.0 pps @ 9.4 Mbps
[ 18] 500 Q 875,515 E 124,489 H 18,558 D 15,042 I 10,002 R 1,459 C 1,424 L 28K
*** crawling 69.0 pps @ 8.5 Mbps
[ 20] 500 Q 865,100 E 134,904 H 20,146 D 16,367 I 10,912 R 1,603 C 1,571 L 31K
*** crawling 72.9 pps @ 8.8 Mbps
[ 22] 500 Q 853,986 E 146,018 H 21,885 D 17,794 I 11,882 R 1,736 C 1,711 L 34K
*** crawling 69.5 pps @ 8.9 Mbps
[ 24] 500 Q 843,237 E 156,767 H 23,564 D 19,194 I 12,809 R 1,880 C 1,853 L 37K
*** crawling 70.4 pps @ 17.0 Mbps
[ 26] 500 Q 831,295 E 168,709 H 25,203 D 20,546 I 13,686 R 2,003 C 1,981 L 39K
*** crawling 64.0 pps @ 12.3 Mbps
[ 28] 500 Q 817,621 E 182,383 H 26,773 D 21,865 I 14,562 R 2,134 C 2,110 L 42K
*** crawling 64.0 pps @ 12.6 Mbps
[ 30] 500 Q 805,730 E 194,274 H 28,282 D 23,111 I 15,399 R 2,251 C 2,219 L 43K
*** crawling 54.5 pps @ 5.9 Mbps
[ 32] 500 Q 794,256 E 205,748 H 29,850 D 24,427 I 16,278 R 2,370 C 2,352 L 46K
*** crawling 66.0 pps @ 12.6 Mbps
[ 34] 500 Q 781,487 E 218,517 H 31,682 D 25,955 I 17,263 R 2,498 C 2,482 L 47K
*** crawling 65.0 pps @ 8.5 Mbps
[ 36] 500 Q 767,190 E 232,814 H 33,491 D 27,434 I 18,203 R 2,627 C 2,604 L 49K
*** crawling 60.5 pps @ 7.1 Mbps
[ 38] 500 Q 756,869 E 243,135 H 35,106 D 28,791 I 19,103 R 2,759 C 2,729 L 50K
*** crawling 62.5 pps @ 7.1 Mbps
[ 40] 500 Q 746,922 E 253,082 H 36,812 D 30,215 I 20,042 R 2,887 C 2,868 L 53K
*** crawling 68.9 pps @ 10.0 Mbps
[ 42] 500 Q 736,268 E 263,736 H 38,528 D 31,667 I 20,976 R 3,018 C 2,991 L 55K
*** crawling 61.5 pps @ 10.3 Mbps
[ 44] 500 Q 726,929 E 273,075 H 40,139 D 33,020 I 21,843 R 3,143 C 3,104 L 56K
*** crawling 56.1 pps @ 7.2 Mbps
[ 46] 500 Q 715,803 E 284,201 H 42,023 D 34,626 I 22,851 R 3,288 C 3,258 L 59K
*** crawling 77.0 pps @ 9.5 Mbps
[ 48] 500 Q 706,585 E 293,419 H 43,681 D 36,012 I 23,729 R 3,409 C 3,378 L 60K
*** crawling 60.0 pps @ 5.9 Mbps
[ 50] 500 Q 696,800 E 303,204 H 45,368 D 37,432 I 24,615 R 3,554 C 3,519 L 62K
*** crawling 69.9 pps @ 7.8 Mbps
[ 52] 500 Q 687,947 E 312,057 H 46,874 D 38,692 I 25,440 R 3,679 C 3,649 L 65K
*** crawling 65.0 pps @ 9.1 Mbps
[ 54] 500 Q 677,627 E 322,377 H 48,469 D 40,021 I 26,280 R 3,813 C 3,780 L 67K
*** crawling 65.0 pps @ 12.2 Mbps
[ 56] 500 Q 667,261 E 332,743 H 50,221 D 41,501 I 27,226 R 3,933 C 3,899 L 69K
*** crawling 59.5 pps @ 7.4 Mbps
[ 58] 500 Q 658,124 E 341,880 H 51,833 D 42,859 I 28,052 R 4,049 C 4,022 L 71K
*** crawling 61.5 pps @ 6.6 Mbps
[ 60] 500 Q 648,183 E 351,821 H 53,339 D 44,126 I 28,850 R 4,151 C 4,120 L 73K
*** crawling 48.6 pps @ 6.1 Mbps
[ 62] 500 Q 635,637 E 364,367 H 55,133 D 45,634 I 29,812 R 4,302 C 4,258 L 75K
*** crawling 68.5 pps @ 10.6 Mbps
[ 64] 500 Q 622,582 E 377,422 H 56,700 D 46,938 I 30,645 R 4,439 C 4,400 L 78K
*** crawling 71.0 pps @ 8.4 Mbps
[ 66] 500 Q 608,986 E 391,018 H 58,400 D 48,383 I 31,527 R 4,573 C 4,526 L 79K
*** crawling 62.5 pps @ 6.7 Mbps
[ 68] 500 Q 593,462 E 406,542 H 60,200 D 49,890 I 32,446 R 4,706 C 4,666 L 82K
*** crawling 69.4 pps @ 14.2 Mbps
[ 70] 500 Q 579,931 E 420,073 H 61,870 D 51,283 I 33,283 R 4,827 C 4,787 L 85K
*** crawling 60.0 pps @ 12.2 Mbps
[ 72] 500 Q 567,601 E 432,403 H 63,515 D 52,656 I 34,096 R 4,952 C 4,904 L 86K
*** crawling 58.5 pps @ 8.8 Mbps
[ 74] 500 Q 555,108 E 444,896 H 65,230 D 54,088 I 34,943 R 5,057 C 5,013 L 88K
*** crawling 54.1 pps @ 7.5 Mbps
[ 76] 500 Q 544,011 E 455,993 H 66,822 D 55,397 I 35,735 R 5,162 C 5,118 L 92K
*** crawling 52.1 pps @ 14.5 Mbps
[ 78] 500 Q 529,584 E 470,420 H 68,700 D 56,956 I 36,669 R 5,300 C 5,255 L 95K
*** crawling 68.5 pps @ 8.9 Mbps
[ 80] 500 Q 514,971 E 485,033 H 70,395 D 58,372 I 37,484 R 5,427 C 5,386 L 98K
*** crawling 65.0 pps @ 9.6 Mbps
[ 82] 500 Q 497,388 E 502,616 H 71,981 D 59,697 I 38,257 R 5,550 C 5,508 L 99K
*** crawling 61.0 pps @ 7.7 Mbps
[ 84] 500 Q 481,043 E 518,961 H 73,502 D 60,982 I 38,995 R 5,646 C 5,599 L 100K
*** crawling 45.5 pps @ 5.6 Mbps
[ 86] 500 Q 467,907 E 532,097 H 75,167 D 62,391 I 39,788 R 5,765 C 5,725 L 103K
*** crawling 62.5 pps @ 9.2 Mbps
[ 88] 500 Q 454,363 E 545,641 H 76,796 D 63,784 I 40,593 R 5,875 C 5,831 L 104K
*** crawling 53.0 pps @ 8.5 Mbps
[ 90] 500 Q 443,161 E 556,843 H 78,381 D 65,106 I 41,354 R 5,986 C 5,945 L 106K
*** crawling 56.5 pps @ 6.1 Mbps
[ 92] 500 Q 432,797 E 567,207 H 80,055 D 66,447 I 42,067 R 6,091 C 6,047 L 114K
*** crawling 50.6 pps @ 7.3 Mbps
[ 94] 500 Q 422,132 E 577,872 H 81,755 D 67,769 I 42,724 R 6,200 C 6,153 L 116K
*** crawling 52.6 pps @ 6.3 Mbps
[ 96] 500 Q 410,641 E 589,363 H 83,302 D 69,004 I 43,320 R 6,280 C 6,229 L 118K
*** crawling 38.0 pps @ 4.6 Mbps
[ 98] 500 Q 396,792 E 603,212 H 85,230 D 70,561 I 44,018 R 6,411 C 6,355 L 120K
*** crawling 62.5 pps @ 9.8 Mbps
[100] 500 Q 379,915 E 620,089 H 87,599 D 72,528 I 44,797 R 6,522 C 6,472 L 122K
*** crawling 58.1 pps @ 10.0 Mbps
[102] 500 Q 361,933 E 638,071 H 89,943 D 74,420 I 45,518 R 6,626 C 6,577 L 125K
*** crawling 52.5 pps @ 8.3 Mbps
[104] 500 Q 345,907 E 654,097 H 92,181 D 76,198 I 46,200 R 6,727 C 6,672 L 128K
*** crawling 47.5 pps @ 5.4 Mbps
[106] 500 Q 330,809 E 669,202 H 94,401 D 77,987 I 46,832 R 6,822 C 6,773 L 130K
*** crawling 50.1 pps @ 7.3 Mbps
[108] 500 Q 314,768 E 685,237 H 96,772 D 79,884 I 47,531 R 6,927 C 6,873 L 131K
*** crawling 50.0 pps @ 9.1 Mbps
[110] 500 Q 299,514 E 700,490 H 99,001 D 81,668 I 48,203 R 7,038 C 6,978 L 133K
*** crawling 52.1 pps @ 11.3 Mbps
[112] 500 Q 281,480 E 718,524 H 101,635 D 83,770 I 48,952 R 7,165 C 7,112 L 135K
*** crawling 66.5 pps @ 6.3 Mbps
[114] 500 Q 263,294 E 736,710 H 104,153 D 85,807 I 49,625 R 7,270 C 7,218 L 139K
*** crawling 52.6 pps @ 6.2 Mbps
[116] 500 Q 246,073 E 753,931 H 106,562 D 87,727 I 50,292 R 7,384 C 7,332 L 142K
*** crawling 56.6 pps @ 8.5 Mbps
[118] 500 Q 230,142 E 769,862 H 108,858 D 89,614 I 50,925 R 7,498 C 7,444 L 144K
*** crawling 55.6 pps @ 7.3 Mbps
[120] 500 Q 212,264 E 787,740 H 111,705 D 91,921 I 51,665 R 7,610 C 7,549 L 146K
*** crawling 52.1 pps @ 7.1 Mbps
[122] 500 Q 192,762 E 807,242 H 114,369 D 94,086 I 52,377 R 7,719 C 7,663 L 148K
*** crawling 57.0 pps @ 8.4 Mbps
[124] 500 Q 177,037 E 822,967 H 116,729 D 95,976 I 53,010 R 7,822 C 7,760 L 150K
*** crawling 48.1 pps @ 5.6 Mbps
[126] 500 Q 159,881 E 840,123 H 119,201 D 97,938 I 53,673 R 7,926 C 7,870 L 151K
*** crawling 54.6 pps @ 7.8 Mbps
[128] 500 Q 144,666 E 855,338 H 121,217 D 99,507 I 54,220 R 8,004 C 7,949 L 153K
*** crawling 39.5 pps @ 4.2 Mbps
[130] 500 Q 132,008 E 867,996 H 122,832 D 100,724 I 54,641 R 8,073 C 8,018 L 154K
*** crawling 34.2 pps @ 4.9 Mbps
[132] 500 Q 118,440 E 881,564 H 124,544 D 102,034 I 55,090 R 8,133 C 8,081 L 155K
*** crawling 31.2 pps @ 4.7 Mbps
[134] 500 Q 104,704 E 895,300 H 126,509 D 103,554 I 55,627 R 8,220 C 8,166 L 156K
*** crawling 42.5 pps @ 4.3 Mbps
[136] 500 Q 90,261 E 909,743 H 128,676 D 105,294 I 56,244 R 8,308 C 8,257 L 157K
*** crawling 45.5 pps @ 5.9 Mbps
[138] 500 Q 72,537 E 927,467 H 131,416 D 107,446 I 56,940 R 8,412 C 8,355 L 159K
*** crawling 49.0 pps @ 6.2 Mbps
[140] 500 Q 53,354 E 946,650 H 134,058 D 109,531 I 57,641 R 8,520 C 8,461 L 162K
*** crawling 52.6 pps @ 7.1 Mbps
[142] 500 Q 36,467 E 963,537 H 136,286 D 111,239 I 58,228 R 8,613 C 8,553 L 164K
*** crawling 45.7 pps @ 6.5 Mbps
[144] 500 Q 17,471 E 982,533 H 138,545 D 112,991 I 58,823 R 8,715 C 8,657 L 166K
*** crawling 51.6 pps @ 6.6 Mbps
[146] 328 Q 0 E 1,000,004 H 139,300 D 113,589 I 59,013 R 8,754 C 8,708 L 169K
*** crawling 25.3 pps @ 5.8 Mbps
[148] 194 Q 0 E 1,000,004 H 139,300 D 113,590 I 59,013 R 8,754 C 8,708 L 169K
*** crawling 0.0 pps @ 0.0 Mbps
[150] 135 Q 0 E 1,000,004 H 139,300 D 113,590 I 59,013 R 8,754 C 8,708 L 169K
*** crawling 0.0 pps @ 0.0 Mbps
[152] 68 Q 0 E 1,000,004 H 139,300 D 113,590 I 59,013 R 8,754 C 8,708 L 169K
*** crawling 0.0 pps @ 0.0 Mbps
[154] 25 Q 0 E 1,000,004 H 139,300 D 113,590 I 59,013 R 8,754 C 8,708 L 169K
*** crawling 0.0 pps @ 0.0 Mbps
Extracted 1,000,004 URLs @ 6,397/s
Looked up 139,300 DNS names @ 891/s
Attempted 59,013 site robots @ 377/s
Crawled 8,708 pages @ 55/s (134.66 MB)
Parsed 169,311 links @ 1,083/s
HTTP codes: 2xx = 3,730, 3xx = 866, 4xx = 4,019, 5xx = 93, other = 0
Report data: elapsed = 156.31 s, page bytes = 134,658,927, robots bytes = 20,451,114, robots 4xx = 8,754, average = 7.94 Mbps
TAMU source pages = 1, outside-TAMU source pages = 00 views