I normally test out a django project using a local vagrant instance. Vagrant creates a virtual machine running the django project which instantly recognizes changes to code. However, I was running a test recently and the VM suddenly started outputting error messages as follows.
Related Articles
(Scroll to the bottom of this post to skip to the solution)
(webapp) [vagrant@project vagrant]$ ./manage.py test
Traceback (most recent call last):
File "./manage.py", line 8, in <module>
execute_from_command_line(sys.argv)
File "/home/vagrant/venvs/webapp/lib/python2.7/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
File "/home/vagrant/venvs/webapp/lib/python2.7/site-packages/django/core/management/__init__.py", line 345, in execute
File "/home/vagrant/venvs/webapp/lib/python2.7/site-packages/django/core/management/commands/test.py", line 30, in run_from_argv
File "/home/vagrant/venvs/webapp/lib/python2.7/site-packages/django/core/management/base.py", line 348, in run_from_argv
File "/home/vagrant/venvs/webapp/lib/python2.7/site-packages/django/core/management/commands/test.py", line 74, in execute
File "/home/vagrant/venvs/webapp/lib/python2.7/site-packages/raven/contrib/django/management/__init__.py", line 41, in new_execute
File "/home/vagrant/venvs/webapp/lib/python2.7/site-packages/django/core/management/base.py", line 413, in execute
File "/home/vagrant/venvs/webapp/lib/python2.7/site-packages/django/utils/translation/__init__.py", line 147, in activate
File "/home/vagrant/venvs/webapp/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 216, in activate
File "/home/vagrant/venvs/webapp/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 205, in translation
File "/home/vagrant/venvs/webapp/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 118, in __init__
IOError: No translation files found for default language en-us.No translation files? I’ve never heard of them. Thinking it could be a random error, I tried to run the test again and got:
(webapp) [vagrant@project vagrant]$ ./manage.py test -bash: ./manage.py: /usr/bin/env: bad interpreter: Input/output error
I tried logging out of the instance and got an I/O error:
(webapp) [vagrant@project vagrant]$ logout -bash: /home/vagrant/.bash_logout: Input/output error
Attempting to ssh into the VM again is refused:
MR-PC:project mruttley$ vagrant ssh ssh_exchange_identification: Connection closed by remote host
Sometimes the “turning it off and on again” solution can work, so let’s try vagrant reload:
MR-PC:project mruttley$ vagrant reload
==> default: Attempting graceful shutdown of VM...
default: Guest communication could not be established! This is usually because
default: SSH is not running, the authentication information was changed,
default: or some other networking issue. Vagrant will force halt, if
default: capable.
==> default: Forcing shutdown of VM...
==> default: Checking if box 'bento/centos-7.1' is up to date...
==> default: Clearing any previously set forwarded ports...
==> default: Fixed port collision for 22 => 2222. Now on port 2200.
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
default: Adapter 2: hostonly
==> default: Forwarding ports...
default: 22 => 2200 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 127.0.0.1:2200
default: SSH username: vagrant
default: SSH auth method: private key
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
^C==> default: Waiting for cleanup before exiting...
MR-PC:project mruttley$ Vagrant exited after cleanup due to external interrupt.A timeout error this round. The error seems related to authentication, but it’s not the whole story. The VM is running according to global-status, despite not being correctly set up, which is a bit strange.
MR-PC:project mruttley$ vagrant global-status id name provider state directory ---------------------------------------------------------------------------------------- 0e8faad default virtualbox running /Volumes/Data/Users/mruttley/Documents/code/project
I also run a GUI called VirtualBox which is sometimes handy for visualizing all VMs on your laptop. Checking there, it also seems to be present – so global-status wasn’t lying to us. Let’s try halting it:
MR-PC:project mruttley$ vagrant halt
==> default: Attempting graceful shutdown of VM...
default: Guest communication could not be established! This is usually because
default: SSH is not running, the authentication information was changed,
default: or some other networking issue. Vagrant will force halt, if
default: capable.
==> default: Forcing shutdown of VM...A forced shutdown this time. So far debugging this error isn’t going well. It is totally strange because the VM was working perfectly beforehand. Vagrant up after this produces the same timeouts:
MR-PC:project mruttley$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Checking if box 'bento/centos-7.1' is up to date...
==> default: Clearing any previously set forwarded ports...
==> default: Fixed port collision for 22 => 2222. Now on port 2200.
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
default: Adapter 2: hostonly
==> default: Forwarding ports...
default: 22 => 2200 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 127.0.0.1:2200
default: SSH username: vagrant
default: SSH auth method: private key
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...Let’s try destroying the vagrant instance completely….
MR-PC:project mruttley$ vagrant destroy
default: Are you sure you want to destroy the 'default' VM? [y/N] y
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...
==> default: Running cleanup tasks for 'ansible' provisioner...
==> default: Running cleanup tasks for 'hostsupdate' provisioner...And starting from scratch:
MR-PC:project mruttley$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'bento/centos-7.1'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'bento/centos-7.1' is up to date...
==> default: Setting the name of the VM: project
==> default: Fixed port collision for 22 => 2222. Now on port 2200.
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
default: Adapter 2: hostonly
==> default: Forwarding ports...
default: 22 => 2200 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 127.0.0.1:2200
default: SSH username: vagrant
default: SSH auth method: private key
default: Warning: Connection timeout. Retrying...
default:
default: Vagrant insecure key detected. Vagrant will automatically replace
default: this with a newly generated keypair for better security.
default:
default: Inserting generated public key within guest...
default: Removing insecure key from the guest if it's present...
default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Configuring and enabling network interfaces...
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!
/sbin/ifup enp0s8
Stdout from the command:
ERROR : [/etc/sysconfig/network-scripts/ifup-eth] Error, some other host already uses address 192.168.111.224.
Stderr from the command:
That’s a very weird error. Perhaps something failed randomly?
MR-PC:project mruttley$ vagrant up Bringing machine 'default' up with 'virtualbox' provider... ==> default: Checking if box 'bento/centos-7.1' is up to date... ==> default: VirtualBox VM is already running.
Apparently it still created the VM. SSHing into it gives this:
MR-PC:project mruttley$ vagrant ssh [vagrant@localhost ~]$ [vagrant@localhost ~]$ ls [vagrant@localhost ~]$ [vagrant@localhost ~]$ [vagrant@localhost ~]$ logout Connection to 127.0.0.1 closed.
Bizarrely there are no files! Something still didn’t startup properly.
[vagrant@localhost ~]$ cd / [vagrant@localhost /]$ ls bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
At this point I was getting a bit frustrated and started to Google the error. The most relevant blog post I could find was Mike Berggren’s solution here: http://mikeberggren.com/post/100289806126/if-up. He fixed this issue and reports: “I’ll spare you the rest of the gory details but suffice it to say, we eventually circumvented that check and ran that same command from the host. It came back with another MAC address claiming ownership“. Does this mean he literally commented out the lines of code that make that check? He may have had a very different problem and I’m no dev-ops expert but perhaps there’s a better solution – maybe that check is there for a reason.
Let’s look back at that network error mentioned before:
The following SSH command responded with a non-zero exit status. Vagrant assumes that this means the command failed! /sbin/ifup enp0s8 Stdout from the command: ERROR : [/etc/sysconfig/network-scripts/ifup-eth] Error, some other host already uses address 192.168.111.224. Stderr from the command:
Cat the vagrant file and it reveals the same IP address. This means that this particular vagrant is always assigned that IP address. If this is the only instance, how is it that something else on this virtual network is using it? Something inside my mac is conflicting with it.
MR-PC:project mruttley$ cat Vagrantfile | grep 111.224 config.vm.network :private_network, ip: "192.168.111.224"
We can check to see what’s really running (aside from what global-status and VirtualBox say) by checking running processes:
MR-PC:project mruttley$ ps aux | grep -i vbox | grep project mruttley 67692 1.0 6.0 3077812 502624 ?? S 22Mar16 87:35.54 /Applications/VirtualBox.app/Contents/MacOS/VBoxHeadless --comment project --startvm (hash) --vrde config mruttley 21971 0.6 5.5 3023580 459532 ?? S 5:22PM 1:00.31 /Applications/VirtualBox.app/Contents/MacOS/VBoxHeadless --comment project --startvm (hash) --vrde config
Aha! There are actually 2 of them. Let’s destroy the non-functioning blank VM that we created before and see if anything has changed:
MR-PC:project mruttley$ vagrant destroy
default: Are you sure you want to destroy the 'default' VM? [y/N] y
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...
==> default: Running cleanup tasks for 'ansible' provisioner...
==> default: Running cleanup tasks for 'hostsupdate' provisioner...Yup, it has gone:
MR-PC:project mruttley$ ps aux | grep -i vbox mruttley 67692 1.3 6.0 3077812 502640 ?? R 22Mar16 87:35.98 /Applications/VirtualBox.app/Contents/MacOS/VBoxHeadless --comment project --startvm (hash) --vrde config
Use kill -9 [pid] to remove it.
MR-PC:project mruttley$ kill -9 67692
Now let’s try recreating the VM:
MR-PC:project mruttley$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'bento/centos-7.1'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'bento/centos-7.1' is up to date...
==> default: Setting the name of the VM: project
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
default: Adapter 2: hostonly
==> default: Forwarding ports...
default: 22 => 2222 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 127.0.0.1:2222
default: SSH username: vagrant
default: SSH auth method: private key
default: Warning: Connection timeout. Retrying...
default: Warning: Remote connection disconnect. Retrying...
default:
default: Vagrant insecure key detected. Vagrant will automatically replace
default: this with a newly generated keypair for better security.
default:
default: Inserting generated public key within guest...
default: Removing insecure key from the guest if it's present...
default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Configuring and enabling network interfaces...
==> default: Mounting shared folders...
default: /vagrant => /Volumes/Data/Users/mruttley/Documents/code/project
==> default: Running provisioner: ansible...It works!
Conclusions:
- I’m still not sure why this happened in the first place. The errors at the start came totally out of the blue – I’d been running that VM for several weeks without issues. Perhaps it was the fact that I’d been running it for so long?
- Vagrant global-status and VirtualBox seem to not entirely accurately report running VMs so definitely check all running vbox processes using ps and remove any extras that didn’t shut down properly. This reminds me of the –prune option in global-status (https://www.vagrantup.com/docs/cli/global-status.html) which can fix persistent old entries.
This post first appeared on Ikigomu | A Data Science, NLP And Personal Blog By, please read the originial post: here
