Discovery

The Python Infrastructure uses Consul to implement service discovery. There is an agent running in client mode on all servers in the iad1 datacenter. These can be queried on the standard ports on localhost. Consul also has a web interface running on localhost on all nodes.

Using the Web Interface

To use the Consul web interface simply ssh into any server with a SSH port forward such as ssh salt-master.psf.io -L 8500:localhost:8500 and then navigate to http://localhost:8500/ in your browser.

Using a Service

Generally the best way to handle pointing to a service is to utilize the consul-template utility which will automatically watch a service and update a configuration file and reload/restart a process whenever it is updated. This takes a template (which can be rendered by salt) and will detect which services it needs based on that.

This is best demonstrated by an example, say haproxy:

haproxy:
  service.running:
    - enable: True
    - reload: True
    - require:
      - service: haproxy-consul

/etc/haproxy/haproxy.cfg.tmpl:
  file.managed:
    - source: salt://haproxy/config/haproxy.cfg.jinja
    - template: jinja

haproxy-consul:
  file.managed:
    - name: /etc/init/haproxy-consul.conf
    - source: salt://consul/init/consul-template.conf.jinja
    - template: jinja
    - context:
        templates:
          - "/etc/haproxy/haproxy.cfg.tmpl:/etc/haproxy/haproxy.cfg:chmod 644 /etc/haproxy/haproxy.cfg && service haproxy reload"
    - require:
      - pkg: consul-pkgs

  service.running:
    - enable: True
    - restart: True
    - require:
      - pkg: consul-pkgs
    - watch:
      - file: haproxy-consul
      - file: /etc/consul-template.conf
      - file: /etc/haproxy/haproxy.cfg.tmpl

Essentially this sets up a haproxy proxy service which requires the haproxy-consul service. Then it renders a configuration template and finally it creates a haproxy-consul instance which has an upstart config and is set to running. The templates variable passed into the contents is important. It is a list of template, destiniation, command tuples. This is in the form of "/path/to/template:/path/to/destination:shell command to run".

The template file can contain blocks that look like:

{{range service "my-service@iad1"}}
server {{.Name}} {{.Address}}:{{.Port}} check{{end}}

This will render a server line for every entry in the "my-service" service in the iad1 datacenter and tt will use the name, address, and port of the registered service. For a full list of everything you can do in this template file take a look at the consul-template documentation.

Registering a Service

Registering an internal service (e.g. one that is running on a server in the iad1 datacenter) is quite easy. You simply need to pick a name, port, and any tags that you wish to associate with the service. Then simply add the below block to the state files:

/etc/consul.d/service-my-service.json:
  file.managed:
    - source: salt://consul/etc/service.jinja
    - template: jinja
    - context:
        name: my-service
        port: 9000
        tags:
          - tag1
          - tag2
    - user: root
    - group: root
    - mode: 644
    - require:
      - pkg: consul-pkgs

Where the name, port, and tags context variables control the values that will be entered into the system. This will be available the next time that salt runs the highstate command. It is likely you’ll want this state to require whatever states setup the service that you’re exposing as any watchers will start using it near instantly.

Registering an External Service

Not all services are hosted internally, some services are external services where we cannot install a consul client on their servers. The Consul service system can handle this quite easily as well. To add an external service simply edit pillar/dev/consul.sls or pillar/prod/consul.sls and add a new entry in the external dictionary. The keys are datacenter, node, address, service, port. Using an external service is exactly like using an internal service.

Example:

consul:
  external:
    - datacenter: vagrant
      node: pythonanywhere
      address: www.pythonanywhere.com
      service: console
      port: 443