EC2 scraper instance #137
EC2 scraper instance #137
Conversation
"Network interfaces and an instance-level security groups may not be specified on the same request"
Can't use instance storage on tiny instance types, plus I should probably use 16.04 LTS over 14.04 LTS at this point.
| namespace :aws do | ||
| region = 'us-east-1' | ||
| availability_zone = 'us-east-1b' | ||
| key_name = 'erickey' |
There was a problem hiding this comment.
¯_(ツ)_/¯ I copied it from the ASG
|
I'm cool with this, and so glad you're making progress on it. My one wish (and it's not a blocker) is to avoid relying on system Python, Ruby, and Node, rather than user-space language managers that stay current with their communities. Thank you for tackling this, and let me know how I can support the work. |
|
Thanks for the guide, I'll take a look once I get this working. |
Due to deprecation warning
The "AWS SDK for Ruby Developer Guide" appears to be wrong/out of date
|
FWIW I ran into this issue: aws/aws-sdk-ruby#859, going to add a hardcoded delay to smooth over the race condition. |
|
I got most of the configuration working, next up tomorrow is figuring out authorization for the AWS managed elasticsearch. |
|
This is coming along pretty well so far, most things on the scraper instance are in working order. My next plan is to make a private S3 bucket with a script in it to add secrets to the configuration file, (Slack, dashboard, etc.) and then grab and invoke the script during setup. |
|
Also, I'm going to do further research on AWS ES authentication libraries. I'm not happy with the one I have in there now, as it requires extra callbacks everywhere. |
|
I've been taking a look at rbenv/pyenv/nvm, and while all the instructions are for adding these tools to .bashrc, that doesn't apply to non-interactive scripts such as the user-data script or the crontab. We already use the below snippet for rbenv, @konklone do you have corresponding snippets for nvm and pyenv? Any other advice for handling this? |
|
I think I have rbenv/pyenv/nvm working now, waiting for the cron job to run to confirm. |
|
Alright, this works now! Ready for review, and I'll follow up with the web server elsewhere. |
|
I'll work on reviewing this this weekend. Thank you for doing all of this! |
Conflicts: npm-shrinkwrap.json package.json
|
|
||
| sudo -u ubuntu git clone https://github.com/sstephenson/rbenv.git /home/ubuntu/.rbenv | ||
| sudo -u ubuntu git clone https://github.com/sstephenson/ruby-build.git /home/ubuntu/.rbenv/plugins/ruby-build | ||
| echo 'export PATH=$HOME/.rbenv/bin:$PATH' | sudo -u ubuntu tee -a /home/ubuntu/.bashrc |
There was a problem hiding this comment.
These commands use sudo, but the ones above don't. If this is running as root, then it won't matter, but just flagging in case the inconsistency is important.
There was a problem hiding this comment.
Yes, the user-data script does run as root
|
Ugh, I'm clearly not prioritizing review here, and don't see a path to reviewing the actual deployment over the next two weeks. I've 👀'd the code, and it looks fine, and I think I just need to 👍 you rolling this out to see how it works in practice. Are you up for that? Thank you so much for doing all this great (and very necessary) configuration management work. |
|
Sure, sounds good. I left the scraper instance running, and it appears to be chugging along just fine. |
This is part of #66. Haven't tested it yet, but it should hit all the high parts!