diff --git a/.gitignore b/.gitignore index 11ac852..ae88bda 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ *.DS_STORE -node_modules \ No newline at end of file +node_modules +views/partials/md/ \ No newline at end of file diff --git a/md_writeup/decaptcha-structure.md b/md_writeup/decaptcha-structure.md new file mode 100644 index 0000000..051ec08 --- /dev/null +++ b/md_writeup/decaptcha-structure.md @@ -0,0 +1,40 @@ +WADDUP MA BRO The 'decaptcha' system is an appendage to the [frontend](frontend-structure.md). It deals specifically with preparing Craigslist ads extracted by the fronend to be processed by [stager](stager-structure.md). In order for a listing to be processed by stager it needs to have some form of contact information. Craigslist hides contact information for listings behind [CAPTCHA's](https://en.wikipedia.org/wiki/CAPTCHA). Decaptcha automates a method to circumvent this and scrape the required contact info. + + + +The above *private* video displays in gui-form how the current decaptcha process works. At present, Craigslist does not require captcha solutions from the user, but still hides the contact information behind an [invisible reCaptcha](https://developers.google.com/recaptcha/docs/invisible). This means that allowing the recaptcha javascript to run through an emulated browser is enough to circumvent the captcha. + +## Process + +While Craigslist may change their Captcha solution from time to time, the decaptcha process is generally the same: + +1. The frontend scrape flags ads that were unable to load contact information. This flag typically takes the form of a custom 'stage' column value (ie: for Craigslist, stage is set to 'CL Lookup unassigned'). Some other information about the Ad is also taken. + +2. The decaptcha system queries the database and finds ads to be processed(if any). Ads to run are prioritized by the liklihood of them having good contact info. For Craigslist, this means that ads with a 'Show Contact Info' button are run first, since there is always good contact info behind this button. + +3. The decaptcha system loads the main details page of each ad. The idea is to emulate how a user would interact with the page as much as possible, therefore the contact info url is not directly loaded. + +4. If a reCaptcha is encountered ("I'm not a robot checkbox"), then the relevant information is collected from the page, and a solution request is sent to [2Captcha](https://2captcha.com/). A valid solution request to the (current) 2Captcha API is shown below. + +5. Decaptcha then waits for 2Captcha to solve the request and input the solution. If an invisible reCaptcha was encountered, decaptcha waits for it to resolve itself. + +6. On completion, relevant information is collected, and the ads database entry is updated (including a stage value saying it's good to go). + +## Historical Note + +Historically, this entire process was handled by [casperJS](http://casperjs.org/)(a navigation scripting framework built atop [phantomJS](http://phantomjs.org/)), and [2Captcha](https://2captcha.com/). Requesting a captcha solution from 2Captcha was as easy as the HTTP request shown below: + +```javascript +var params = { + method: userrecaptcha, + key: USER_API_KEY, + googlekey: CRAIGSLIST_SITE_KEY, + pageurl: REQUEST_URL--REPLY_BTN_or_SHOW_CONTACT_BTN, + json: '1' +}; +var url = 'http://2captcha.com/in.php?' + params; +``` + +However casperJS' browser emulation proved troublesome with the latest invisible reCaptcha on Craigslist. In addition, 2Captcha was always seen as a necessary evil. Solutions from 2Captcha were generated by human workers and could take upwards of 60 seconds for a single listing. This made decaptcha the slowest process in the entire ByOwner pipeline. While 2Captcha was a reliable and workable solution, avoidance was optimal. + +These issues presented an opportunity to move the entire decaptcha system into the Java stack already built for ByOwner. The latest solution uses [selenium](http://www.seleniumhq.org/): a multi-language api that plugs into various browser drivers(Chrome, Firefox) and runs them programmatically. This solution allows very user-like scraping, with similar or better resource utilization compared to casperJS. Given that this solution also avoids 2Captcha, process speed is up 2-4x. \ No newline at end of file diff --git a/md_writeup/index.md b/md_writeup/index.md new file mode 100644 index 0000000..c055be2 --- /dev/null +++ b/md_writeup/index.md @@ -0,0 +1,9 @@ +# Welcome to DataLab's ByOwner + +Welcome to the documentation for the Data Laboratory ByOwner project. The purpose of this project is to provide quality real estate data through superior automation technologies. The project is built with java and [screen-scraper](https://www.screen-scraper.com). Data is collected daily from hundreds of sites across North America, sent through rigorous processing, and finally shipped to the end-user. + +## JavaDocs + +The DataLab java project contains all proprietary java code developed by DataLab and is used throughout the entire [ByOwner process](process-structure/overview.md). The following is the javadocs site created for anyone looking to get familiar with/modify the code: + +[JavaDocs for ByOwner](api/) \ No newline at end of file diff --git a/package-lock.json b/package-lock.json index 27c3223..38427f6 100644 --- a/package-lock.json +++ b/package-lock.json @@ -13,6 +13,11 @@ "negotiator": "0.6.1" } }, + "ansi-regex": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-3.0.0.tgz", + "integrity": "sha1-7QMXwyIGT3lGbAKWa922Bas32Zg=" + }, "array-flatten": { "version": "1.1.1", "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", @@ -40,6 +45,26 @@ "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.0.0.tgz", "integrity": "sha1-0ygVQE1olpn4Wk6k+odV3ROpYEg=" }, + "camelcase": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/camelcase/-/camelcase-4.1.0.tgz", + "integrity": "sha1-1UVjW+HjPFQmScaRc+Xeas+uNN0=" + }, + "cliui": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/cliui/-/cliui-4.1.0.tgz", + "integrity": "sha512-4FG+RSG9DL7uEwRUZXZn3SS34DiDPfzP0VOiEwtUWlE+AR2EIg+hSyvrIgUUfhdgR/UkAeW2QHgeP+hWrXs7jQ==", + "requires": { + "string-width": "^2.1.1", + "strip-ansi": "^4.0.0", + "wrap-ansi": "^2.0.0" + } + }, + "code-point-at": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/code-point-at/-/code-point-at-1.1.0.tgz", + "integrity": "sha1-DQcLTQQ6W+ozovGkDi7bPZpMz3c=" + }, "content-disposition": { "version": "0.5.2", "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.2.tgz", @@ -60,6 +85,16 @@ "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.0.6.tgz", "integrity": "sha1-4wOogrNCzD7oylE6eZmXNNqzriw=" }, + "cross-spawn": { + "version": "5.1.0", + "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-5.1.0.tgz", + "integrity": "sha1-6L0O/uWPz/b4+UUQoKVUu/ojVEk=", + "requires": { + "lru-cache": "^4.0.1", + "shebang-command": "^1.2.0", + "which": "^1.2.9" + } + }, "debug": { "version": "2.6.9", "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz", @@ -68,6 +103,11 @@ "ms": "2.0.0" } }, + "decamelize": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/decamelize/-/decamelize-1.2.0.tgz", + "integrity": "sha1-9lNNFRSCabIDUue+4m9QH5oZEpA=" + }, "depd": { "version": "1.1.2", "resolved": "https://registry.npmjs.org/depd/-/depd-1.1.2.tgz", @@ -103,6 +143,20 @@ "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz", "integrity": "sha1-Qa4u62XvpiJorr/qg6x9eSmbCIc=" }, + "execa": { + "version": "0.7.0", + "resolved": "https://registry.npmjs.org/execa/-/execa-0.7.0.tgz", + "integrity": "sha1-lEvs00zEHuMqY6n68nrVpl/Fl3c=", + "requires": { + "cross-spawn": "^5.0.1", + "get-stream": "^3.0.0", + "is-stream": "^1.1.0", + "npm-run-path": "^2.0.0", + "p-finally": "^1.0.0", + "signal-exit": "^3.0.0", + "strip-eof": "^1.0.0" + } + }, "express": { "version": "4.16.3", "resolved": "https://registry.npmjs.org/express/-/express-4.16.3.tgz", @@ -154,6 +208,14 @@ "unpipe": "~1.0.0" } }, + "find-up": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/find-up/-/find-up-2.1.0.tgz", + "integrity": "sha1-RdG35QbHF93UgndaK3eSCjwMV6c=", + "requires": { + "locate-path": "^2.0.0" + } + }, "forwarded": { "version": "0.1.2", "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.1.2.tgz", @@ -164,6 +226,16 @@ "resolved": "https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz", "integrity": "sha1-PYyt2Q2XZWn6g1qx+OSyOhBWBac=" }, + "get-caller-file": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-1.0.3.tgz", + "integrity": "sha512-3t6rVToeoZfYSGd8YoLFR2DJkiQrIiUrGcjvFX2mDw3bn6k2OtwHN0TNCLbBO+w8qTvimhDkv+LSscbJY1vE6w==" + }, + "get-stream": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/get-stream/-/get-stream-3.0.0.tgz", + "integrity": "sha1-jpQ9E1jcN1VQVOy+LtsFqhdO3hQ=" + }, "http-errors": { "version": "1.6.3", "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-1.6.3.tgz", @@ -185,16 +257,70 @@ "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz", "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4=" }, + "invert-kv": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/invert-kv/-/invert-kv-1.0.0.tgz", + "integrity": "sha1-EEqOSqym09jNFXqO+L+rLXo//bY=" + }, "ipaddr.js": { "version": "1.8.0", "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.8.0.tgz", "integrity": "sha1-6qM9bd16zo9/b+DJygRA5wZzix4=" }, + "is-fullwidth-code-point": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-2.0.0.tgz", + "integrity": "sha1-o7MKXE8ZkYMWeqq5O+764937ZU8=" + }, + "is-stream": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/is-stream/-/is-stream-1.1.0.tgz", + "integrity": "sha1-EtSj3U5o4Lec6428hBc66A2RykQ=" + }, + "isexe": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", + "integrity": "sha1-6PvzdNxVb/iUehDcsFctYz8s+hA=" + }, + "lcid": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/lcid/-/lcid-1.0.0.tgz", + "integrity": "sha1-MIrMr6C8SDo4Z7S28rlQYlHRuDU=", + "requires": { + "invert-kv": "^1.0.0" + } + }, + "locate-path": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/locate-path/-/locate-path-2.0.0.tgz", + "integrity": "sha1-K1aLJl7slExtnA3pw9u7ygNUzY4=", + "requires": { + "p-locate": "^2.0.0", + "path-exists": "^3.0.0" + } + }, + "lru-cache": { + "version": "4.1.3", + "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-4.1.3.tgz", + "integrity": "sha512-fFEhvcgzuIoJVUF8fYr5KR0YqxD238zgObTps31YdADwPPAp82a4M8TrckkWyx7ekNlf9aBcVn81cFwwXngrJA==", + "requires": { + "pseudomap": "^1.0.2", + "yallist": "^2.1.2" + } + }, "media-typer": { "version": "0.3.0", "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz", "integrity": "sha1-hxDXrwqmJvj/+hzgAWhUUmMlV0g=" }, + "mem": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/mem/-/mem-1.1.0.tgz", + "integrity": "sha1-Xt1StIXKHZAP5kiVUFOZoN+kX3Y=", + "requires": { + "mimic-fn": "^1.0.0" + } + }, "merge-descriptors": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.1.tgz", @@ -223,6 +349,24 @@ "mime-db": "~1.35.0" } }, + "mimic-fn": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/mimic-fn/-/mimic-fn-1.2.0.tgz", + "integrity": "sha512-jf84uxzwiuiIVKiOLpfYk7N46TSy8ubTonmneY9vrpHNAnp0QBt2BxWV9dO3/j+BoVAb+a5G6YDPW3M5HOdMWQ==" + }, + "minimist": { + "version": "0.0.8", + "resolved": "https://registry.npmjs.org/minimist/-/minimist-0.0.8.tgz", + "integrity": "sha1-hX/Kv8M5fSYluCKCYuhqp6ARsF0=" + }, + "mkdirp": { + "version": "0.5.1", + "resolved": "https://registry.npmjs.org/mkdirp/-/mkdirp-0.5.1.tgz", + "integrity": "sha1-MAV0OOrGz3+MR2fzhkjWaX11yQM=", + "requires": { + "minimist": "0.0.8" + } + }, "ms": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz", @@ -233,6 +377,19 @@ "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.1.tgz", "integrity": "sha1-KzJxhOiZIQEXeyhWP7XnECrNDKk=" }, + "npm-run-path": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/npm-run-path/-/npm-run-path-2.0.2.tgz", + "integrity": "sha1-NakjLfo11wZ7TLLd8jV7GHFTbF8=", + "requires": { + "path-key": "^2.0.0" + } + }, + "number-is-nan": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/number-is-nan/-/number-is-nan-1.0.1.tgz", + "integrity": "sha1-CXtgK1NCKlIsGvuHkDGDNpQaAR0=" + }, "on-finished": { "version": "2.3.0", "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.3.0.tgz", @@ -241,11 +398,57 @@ "ee-first": "1.1.1" } }, + "os-locale": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/os-locale/-/os-locale-2.1.0.tgz", + "integrity": "sha512-3sslG3zJbEYcaC4YVAvDorjGxc7tv6KVATnLPZONiljsUncvihe9BQoVCEs0RZ1kmf4Hk9OBqlZfJZWI4GanKA==", + "requires": { + "execa": "^0.7.0", + "lcid": "^1.0.0", + "mem": "^1.1.0" + } + }, + "p-finally": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/p-finally/-/p-finally-1.0.0.tgz", + "integrity": "sha1-P7z7FbiZpEEjs0ttzBi3JDNqLK4=" + }, + "p-limit": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/p-limit/-/p-limit-1.3.0.tgz", + "integrity": "sha512-vvcXsLAJ9Dr5rQOPk7toZQZJApBl2K4J6dANSsEuh6QI41JYcsS/qhTGa9ErIUUgK3WNQoJYvylxvjqmiqEA9Q==", + "requires": { + "p-try": "^1.0.0" + } + }, + "p-locate": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/p-locate/-/p-locate-2.0.0.tgz", + "integrity": "sha1-IKAQOyIqcMj9OcwuWAaA893l7EM=", + "requires": { + "p-limit": "^1.1.0" + } + }, + "p-try": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/p-try/-/p-try-1.0.0.tgz", + "integrity": "sha1-y8ec26+P1CKOE/Yh8rGiN8GyB7M=" + }, "parseurl": { "version": "1.3.2", "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.2.tgz", "integrity": "sha1-/CidTtiZMRlGDBViUyYs3I3mW/M=" }, + "path-exists": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/path-exists/-/path-exists-3.0.0.tgz", + "integrity": "sha1-zg6+ql94yxiSXqfYENe1mwEP1RU=" + }, + "path-key": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/path-key/-/path-key-2.0.1.tgz", + "integrity": "sha1-QRyttXTFoUDTpLGRDUDYDMn0C0A=" + }, "path-to-regexp": { "version": "0.1.7", "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.7.tgz", @@ -260,6 +463,11 @@ "ipaddr.js": "1.8.0" } }, + "pseudomap": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/pseudomap/-/pseudomap-1.0.2.tgz", + "integrity": "sha1-8FKijacOYYkX7wqKw0wa5aaChrM=" + }, "qs": { "version": "6.5.1", "resolved": "https://registry.npmjs.org/qs/-/qs-6.5.1.tgz", @@ -304,6 +512,16 @@ } } }, + "require-directory": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/require-directory/-/require-directory-2.1.1.tgz", + "integrity": "sha1-jGStX9MNqxyXbiNE/+f3kqam30I=" + }, + "require-main-filename": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/require-main-filename/-/require-main-filename-1.0.1.tgz", + "integrity": "sha1-l/cXtp1IeE9fUmpsWqj/3aBVpNE=" + }, "safe-buffer": { "version": "5.1.1", "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.1.tgz", @@ -340,16 +558,69 @@ "send": "0.16.2" } }, + "set-blocking": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/set-blocking/-/set-blocking-2.0.0.tgz", + "integrity": "sha1-BF+XgtARrppoA93TgrJDkrPYkPc=" + }, "setprototypeof": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.1.0.tgz", "integrity": "sha512-BvE/TwpZX4FXExxOxZyRGQQv651MSwmWKZGqvmPcRIjDqWub67kTKuIMx43cZZrS/cBBzwBcNDWoFxt2XEFIpQ==" }, + "shebang-command": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-1.2.0.tgz", + "integrity": "sha1-RKrGW2lbAzmJaMOfNj/uXer98eo=", + "requires": { + "shebang-regex": "^1.0.0" + } + }, + "shebang-regex": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/shebang-regex/-/shebang-regex-1.0.0.tgz", + "integrity": "sha1-2kL0l0DAtC2yypcoVxyxkMmO/qM=" + }, + "showdown": { + "version": "1.8.6", + "resolved": "https://registry.npmjs.org/showdown/-/showdown-1.8.6.tgz", + "integrity": "sha1-kepO47elRIqspoIKTifmkMatdxw=", + "requires": { + "yargs": "^10.0.3" + } + }, + "signal-exit": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-3.0.2.tgz", + "integrity": "sha1-tf3AjxKH6hF4Yo5BXiUTK3NkbG0=" + }, "statuses": { "version": "1.4.0", "resolved": "https://registry.npmjs.org/statuses/-/statuses-1.4.0.tgz", "integrity": "sha512-zhSCtt8v2NDrRlPQpCNtw/heZLtfUDqxBM1udqikb/Hbk52LK4nQSwr10u77iopCW5LsyHpuXS0GnEc48mLeew==" }, + "string-width": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/string-width/-/string-width-2.1.1.tgz", + "integrity": "sha512-nOqH59deCq9SRHlxq1Aw85Jnt4w6KvLKqWVik6oA9ZklXLNIOlqg4F2yrT1MVaTjAqvVwdfeZ7w7aCvJD7ugkw==", + "requires": { + "is-fullwidth-code-point": "^2.0.0", + "strip-ansi": "^4.0.0" + } + }, + "strip-ansi": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-4.0.0.tgz", + "integrity": "sha1-qEeQIusaw2iocTibY1JixQXuNo8=", + "requires": { + "ansi-regex": "^3.0.0" + } + }, + "strip-eof": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/strip-eof/-/strip-eof-1.0.0.tgz", + "integrity": "sha1-u0P/VZim6wXYm1n80SnJgzE2Br8=" + }, "type-is": { "version": "1.6.16", "resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.16.tgz", @@ -373,6 +644,98 @@ "version": "1.1.2", "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz", "integrity": "sha1-IpnwLG3tMNSllhsLn3RSShj2NPw=" + }, + "which": { + "version": "1.3.1", + "resolved": "https://registry.npmjs.org/which/-/which-1.3.1.tgz", + "integrity": "sha512-HxJdYWq1MTIQbJ3nw0cqssHoTNU267KlrDuGZ1WYlxDStUtKUhOaJmh112/TZmHxxUfuJqPXSOm7tDyas0OSIQ==", + "requires": { + "isexe": "^2.0.0" + } + }, + "which-module": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/which-module/-/which-module-2.0.0.tgz", + "integrity": "sha1-2e8H3Od7mQK4o6j6SzHD4/fm6Ho=" + }, + "wrap-ansi": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-2.1.0.tgz", + "integrity": "sha1-2Pw9KE3QV5T+hJc8rs3Rz4JP3YU=", + "requires": { + "string-width": "^1.0.1", + "strip-ansi": "^3.0.1" + }, + "dependencies": { + "ansi-regex": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-2.1.1.tgz", + "integrity": "sha1-w7M6te42DYbg5ijwRorn7yfWVN8=" + }, + "is-fullwidth-code-point": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-1.0.0.tgz", + "integrity": "sha1-754xOG8DGn8NZDr4L95QxFfvAMs=", + "requires": { + "number-is-nan": "^1.0.0" + } + }, + "string-width": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/string-width/-/string-width-1.0.2.tgz", + "integrity": "sha1-EYvfW4zcUaKn5w0hHgfisLmxB9M=", + "requires": { + "code-point-at": "^1.0.0", + "is-fullwidth-code-point": "^1.0.0", + "strip-ansi": "^3.0.0" + } + }, + "strip-ansi": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-3.0.1.tgz", + "integrity": "sha1-ajhfuIU9lS1f8F0Oiq+UJ43GPc8=", + "requires": { + "ansi-regex": "^2.0.0" + } + } + } + }, + "y18n": { + "version": "3.2.1", + "resolved": "https://registry.npmjs.org/y18n/-/y18n-3.2.1.tgz", + "integrity": "sha1-bRX7qITAhnnA136I53WegR4H+kE=" + }, + "yallist": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/yallist/-/yallist-2.1.2.tgz", + "integrity": "sha1-HBH5IY8HYImkfdUS+TxmmaaoHVI=" + }, + "yargs": { + "version": "10.1.2", + "resolved": "https://registry.npmjs.org/yargs/-/yargs-10.1.2.tgz", + "integrity": "sha512-ivSoxqBGYOqQVruxD35+EyCFDYNEFL/Uo6FcOnz+9xZdZzK0Zzw4r4KhbrME1Oo2gOggwJod2MnsdamSG7H9ig==", + "requires": { + "cliui": "^4.0.0", + "decamelize": "^1.1.1", + "find-up": "^2.1.0", + "get-caller-file": "^1.0.1", + "os-locale": "^2.0.0", + "require-directory": "^2.1.1", + "require-main-filename": "^1.0.1", + "set-blocking": "^2.0.0", + "string-width": "^2.0.0", + "which-module": "^2.0.0", + "y18n": "^3.2.1", + "yargs-parser": "^8.1.0" + } + }, + "yargs-parser": { + "version": "8.1.0", + "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-8.1.0.tgz", + "integrity": "sha512-yP+6QqN8BmrgW2ggLtTbdrOyBNSI7zBa4IykmiV5R1wl1JWNxQvWhMfMdmzIYtKU7oP3OOInY/tl2ov3BDjnJQ==", + "requires": { + "camelcase": "^4.1.0" + } } } } diff --git a/package.json b/package.json index 8ae929f..39ad013 100644 --- a/package.json +++ b/package.json @@ -5,7 +5,7 @@ "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1", - "prestart": "echo waddup" + "prestart": "node prestart.js" }, "repository": { "type": "git", @@ -19,6 +19,8 @@ "homepage": "https://github.com/len0rd/personal-website#readme", "dependencies": { "ejs": "^2.6.1", - "express": "^4.16.3" + "express": "^4.16.3", + "mkdirp": "^0.5.1", + "showdown": "^1.8.6" } } diff --git a/prestart-buildmd.sh b/prestart-buildmd.sh deleted file mode 100644 index c0769a8..0000000 --- a/prestart-buildmd.sh +++ /dev/null @@ -1 +0,0 @@ -echo "Building HTML partials from markdown" \ No newline at end of file diff --git a/prestart.js b/prestart.js new file mode 100644 index 0000000..f128dd2 --- /dev/null +++ b/prestart.js @@ -0,0 +1,42 @@ +// convert our markdown documentation files +// to 'static' html/ejs partials: +// while this is a bit inconvenient (you need to restart +// the server everytime you want to see +// md changes), it is more efficient in +// that we aren't converting MD -> ejs +// on EVERY request +var showdown = require('showdown'), + converter = new showdown.Converter(), + fs = require('fs'), + mkdirp = require('mkdirp'), + inputDir = './md_writeup/', + outputDir = './views/partials/md/'; + +mkdirp(outputDir, function(err){ + if (err) { + console.error(err); + } else { + console.log('output dir created'); + } +}) + +fs.readdir(inputDir, (err, files) => { + files.forEach(file => { + if (file.endsWith('.md')) { + let fileNameNoExtension = file.slice(0, -3); + console.log('converting: ' + fileNameNoExtension); + fs.readFile(inputDir + file, 'utf8', (err, data) => { + if (err) { + console.error(err); + } else { + let html = converter.makeHtml(data); + fs.writeFile(outputDir + fileNameNoExtension + '.ejs', html, 'utf8', (err) => { + if (err) { + console.error(err); + } + }); + } + }); + } + }); +}); diff --git a/server.js b/server.js index 5b649b7..4cee7bf 100644 --- a/server.js +++ b/server.js @@ -2,7 +2,7 @@ const PORT = 8090; var express = require('express'); var app = express(); -console.log('Hello'); +console.log('Starting express server'); // set the view engine to ejs app.set('view engine', 'ejs'); diff --git a/views/pages/index.ejs b/views/pages/index.ejs index 77b235c..156b535 100644 --- a/views/pages/index.ejs +++ b/views/pages/index.ejs @@ -31,7 +31,7 @@ >
- No, my name is not Lenord, but I do enjoy minor internet anonymity, hence the screen name. I'm a Computer Engineering student with diverse industry experience. From large commercial Java projects to low-level C firmware development, I've done a little bit of everything (except web development as will be evident with this website). Such experience allows me to understand and solve new problems quickly. My hobbies are just as eclectic as my professional experience. This website showcases many of my personal projects. + I'm an American Computer Engineering student with diverse industry experience. From large commercial Java projects to low-level C firmware development, I've done a little bit of everything (except web development as will be evident with this website). Such experience allows me to understand and solve new problems quickly. My hobbies are just as eclectic as my professional experience. This website showcases many of my personal projects.