Welcome

What works and what doesn't work in software development? For the first 10 years or so of my career, we followed a strict waterfall development model. Then in 2008 we started switching to an agile development model. In 2011 we added DevOps principles and practices. Our product team has also become increasingly global. This blog is about our successes and failures, and what we've learned along the way.



The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. The comments may not represent my opinions, and certainly do not represent IBM in any way.

Tuesday, August 19, 2014

Food Network's Disturbing Breach of Privacy

I found a recipe on Food Network that I wanted to try later, so I clicked on the "Save Recipe" button. It asked me to log in or create an account, so I decided to try the "Log In With Facebook" option. Here are the permissions it asked for: "Food Network will receive the following info: your public profile, friend list, email address, custom friends lists, birthday, current city, photos and personal description and your friends' religious and political views." No, no, no, no, and no! Food Network, you should be ashamed of yourself.

Friday, August 8, 2014

SoftLayer Gotchas, Tips, and Tricks

NOTE: THIS IS OUTDATED!
Here are many things I learned about SoftLayer the hard way, through trial and error.  Of course, SoftLayer's offerings are evolving, so some of these "facts" will change over time, but reading through this list should still help.

Some of this is RHEL-centric, because I do the majority of my SoftLayer work on RHEL.

Useful links

Support

  • SoftLayer tech support via the online ticketing system is very good.  I open tickets whenever I get stuck, and they get back to me quickly.  They also know what they're talking about; no high school students running the help desk here.

Networking

  • You can't run the SoftLayer SSL VPN and another VPN client  at the same time.  It will seem like it's working, but you won't be able to connect to the SoftLayer systems.
  • Use the SoftLayer private network to administer your systems.  It's more secure and saves money.
  • If you're using the private network, you must have the private network on eth0, and also have a static route: 10.0.0.0/8 via your private subnet gateway server.  If you don't have that static route, you won't be able to connect to other SoftLayer servers on the private network and vice-versa.  SoftLayer compute instances and bare-metal servers have that route by default unless you re-install the OS yourself.  Usually this only happens if you're running a hypervisor within SoftLayer and creating your own VMs there.
  • If you're deploying CCIs or VMs or servers with a public Internet connection, you must never enable root SSH access with an easily guessed id/password, even for a minute!  The machine will be hacked very quickly, by Internet worms that try to log in to all available IP addresses with common IDs and passwords.  Monitor bandwidth usage regularly to ensure that none of the systems are consuming huge amounts of bandwidth; this is a symptom of an Internet worm that has managed to install itself.
  • If you need to ensure that some of your servers are on the same VLAN, use SAN storage, not a local disk, for the primary disk.  If you use a local disk, your server will probably be assigned to a different VLAN, and you won't be able to move it later.  
  • If you want multiple bare-metal servers to be on the same VLAN, order all of them at the same time and be consistent with your storage configuration (local vs. SAN disks).
  • VLAN spanning: By default, servers on different private VLANs can't communicate with each other.  There's a setting where you can enable VLAN spanning, and then all of your private VLANs will be able to communicate with each other (regardless of data center).  
  • When creating new servers, you don't get a choice of subnet (only the VLAN).  If you want to assign servers to specific subnets, request Portable IP Subnets (public or private) and use those instead.  
  • Use the subnet details page (https://control.softlayer.com/network/subnets/*) to "sign out" portable IP addresses by entering the hostnames in the Notes field.  
  • For public IP addresses, you can set up Reverse DNS from the same subnet details page.
  • You can add a dedicated hardware firewall in front of any public subnets, and configure firewall rules.
  • If you want very flexible control over your firewall and VPN access, you can add a Vyatta gateway (or an HA pair of Vyatta gateways) instead.  Each Vyatta gateway can manage 1 "pod", where a pod is all of your own VLANs behind the same router pair (public + private) in the same datacenter.
  • ESX servers will only be connected to the SoftLayer private network.  Their public NICs are disabled.  This was never a problem for us.
  • The private SoftLayer network is not intended for large file transfers.  It is a shared resource, and file transfer speeds can be slow at times.  The public network will give you faster file transfers.

Compute Instances (a.k.a. CCIs)

  • These are essentially VMs running on a Xen hypervisor.
  • If you try to install an unsupported OS (such as RHEL 6.4) on a CCI, you may make the CCI unusable.  You will also not be able to backup and restore the CCI.  It's a bad idea.
  • On Xen, dmidecode doesn't work. 
  • You can't install VMWare drivers on Xen.  You can, however, order a VCenter CCI from SoftLayer and use that to manage your SoftLayer ESX servers.
  • The maximum size for the primary disk is 100 GB.  You can add additional disks if needed.
  • LVM is not supported.  If you try to use LVM to make multiple disks look like a single partition, you may make the CCI unusable.  You will also not be able to back and restore the CCI.  This is also a bad idea.
  • Provisioning time may vary between datacenters. When ordering a CCI, you can select an option to deploy it to the first available datacenter if provisioning speed is more important than where it ends up.

Bare Metal Servers

  • If you install the available AFP firewall, it will probably conflict with iptables unless you know what you're doing.
  • Bare-metal servers are not available with RHEL.  CentOS is available.
  • If you don't see the server configuration you want in the web order form, you may be able to order exactly what you want by starting up a sales chat.
  • If you try to install an unsupported OS (such as RHEL 6.4) on a bare metal server, you will not be able to backup and restore the server.  This is a bad idea.

Red Hat Tips

  • Bare-metal servers are not available with RHEL.  CentOS is available.  Some of our software requires RHEL, and in some cases, we have been able to work around this by installing the prereq packages from the CentOS repo before installing that software.
  • To add a static route to the private network on RHEL/CentOS: add the route to /etc/sysconfig/network-scripts/route-eth0 so the setting will be persisted, and also run "route add -net 10.0.0.0 netmask 255.0.0.0 dev eth0" to enable it immediately.  Google it for details.
  • OS options are somewhat limited; RHEL 6 will always be the latest version of RHEL SoftLayer supports (currently 6.5).
Hope this helps!  I welcome your comments below.