Older blog entries for mikal (starting at number 938)

The Long Earth




ISBN: 9780062068682
LibraryThing
Lana put me onto this book while on a trip to Texas, and I have to say I like it. This is very unlike the other Terry Pratchett books I've read, in that whilst it is occasionally amusing, it isn't really an attempt at humor. It is instead a relatively methodical examination of the impact of discovering a series of inhabitable earths a trivial amount of distance away from our own. I also have to say I like the ending, not in the sense of liking what happens, but in the sense that it wasn't a twee or overly convenient way to stop the book. A good read.

Tags for this post: book terry_pratchett_and_stephen_baxter terry_pratchett stephen_baxter travel population terrorism
Related posts: Caves of Steel; Robots and Empire ; Logan's Run ; Space Soldiers; Robots of Dawn ; Naked Sun


Comment

Syndicated 2014-01-26 17:13:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

Comparing alembic with sqlalchemy migrate

In the last few days there has been a discussion on the openstack-dev mailing list about converting nova to alembic. Nova currently uses sqlalchemy migrate for its schema migrations. I would consider myself a sceptic of this change, but I want to be a well educated sceptic so I thought I should take a look at an existing alembic user, in this case neutron. There is also at least one session on database changes at the Icehouse summit this coming week, and I wanted to feel prepared for those conversations.

I should start off by saying that I'm not particularly opposed to alembic. We definitely have problems with migrate, but I am not sure that these problems are addressed by alembic in the way that we'd hope. I think we need to dig deeper into the issues we face with migrate to understand if alembic is a good choice.

sqlalchemy migrate

There are two problems with migrate that I see us suffering from at the moment. The first is that migrate is no longer maintained by upstream. I can see why this is bad, although there are other nova dependencies that the OpenStack team maintains internally. For example, the various oslo libraries and the oslo incubator. I understand that reducing the amount of code we maintain is good, but migrate is stable and relatively static. Any changes made will be fixes for security issues or feature changes that the OpenStack project wants. This relative stability means that we're unlikely to see gate breakages because of unexpected upstream changes. It also means that when we want to change how migrate works for our convenience, we don't need to spend time selling upstream on that change.

The other problem I see is that its really fiddly to land database migrations in nova at the moment. Migrations are a linear stream though time implemented in the form of a sequential number. So, if the current schema version is 227, then my new migration would be implemented by adding the following files to the git repository:

    184_implement_funky_feature.py
    184_sqlite_downgrade.sql
    184_sqlite_upgrade.sql
    


    In this example, the migration is called "implement_funky_feature", and needs custom sqlite upgrades and downgrades. Those sqlite specific files are optional.

    Now the big problem here is that if there is more than one patch competing for the next migration number (which is quite common), then only one patch can win. The others will need to manually rebase their change by renaming these files and then have to re-attempt the code review process. This is very annoying, especially because migration numbers are baked into our various migration tests.

    "Each" migration also has migration tests, which reside in nova/tests/db/test_migrations.py. I say each in quotes because we haven't been fantastic about actually adding tests for all our migrations, so that is imperfect at best. When you miss out on a migration number, you also need to update your migration tests to have the new version number in them.

    If we ignore alembic for a moment, I think we can address this issue within migrate relatively easily. The biggest problem at the moment is that migration numbers are derived from the file naming scheme. If instead they came from a configuration file, then when you needed to change the migration number for your patch it would be a one line change in a configuration file, instead of a selection of file renames and some changes to tests. Consider a configuration file which looks like this:

      mikal@e7240:~/src/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions$ cat versions.json | head
      {
          "133": [
              "folsom.py"
          ], 
          "134": [
              "add_counters_to_bw_usage_cache.py"
          ], 
          "135": [
              "add_node_to_instances.py"
          ], 
      ...
      


      Here, the only place the version number appears is in this versions.json configuration file. For each version, you just list the files present for the migration. In each of the cases here its just the python migration, but it could just as easily include sqlite specific migrations in the array of filenames.

      Then we just need a very simple change to migrate to prefer the config file if it is present:

        diff --git a/migrate/versioning/version.py b/migrate/versioning/version.py index d5a5be9..cee1e66 100644 --- a/migrate/versioning/version.py +++ b/migrate/versioning/version.py @@ -61,22 +61,31 @@ class Collection(pathed.Pathed): """ super(Collection, self).__init__(path) - # Create temporary list of files, allowing skipped version numbers. - files = os.listdir(path) - if '1' in files: - # deprecation - raise Exception('It looks like you have a repository in the old ' - 'format (with directories for each version). ' - 'Please convert repository before proceeding.') - - tempVersions = dict() - for filename in files: - match = self.FILENAME_WITH_VERSION.match(filename) - if match: - num = int(match.group(1)) - tempVersions.setdefault(num, []).append(filename) - else: - pass # Must be a helper file or something, let's ignore it. + # NOTE(mikal): If there is a versions.json file, use that instead of + # filesystem numbering + json_path = os.path.join(path, 'versions.json') + if os.path.exists(json_path): + with open(json_path) as f: + tempVersions = json.loads(f.read()) + + else: + # Create temporary list of files, allowing skipped version numbers. + files = os.listdir(path) + if '1' in files: + # deprecation + raise Exception('It looks like you have a repository in the ' + 'old format (with directories for each ' + 'version). Please convert repository before ' + 'proceeding.') + + tempVersions = dict() + for filename in files: + match = self.FILENAME_WITH_VERSION.match(filename) + if match: + num = int(match.group(1)) + tempVersions.setdefault(num, []).append(filename) + else: + pass # Must be a helper file or something, let's ignore it. # Create the versions member where the keys # are VerNum's and the values are Version's.


      There are some tweaks required to test_migrations.py as well, but they are equally trivial. As an aside, I wonder what people think about moving the migration tests out of the test tree and into the versions directory so that they are beside the migrations. This would make it clearer which migrations lack tests, and would reduce the length of test_migrations.py, which is starting to get out of hand at 3,478 lines.

      There's one last thing I want to say about migrate migrations before I move onto discussing alembic. One of the features of migrate is that schema migrations are linear, which I think is a feature not a limitation. In the Havana (and presumably Icehouse) releases there has been significant effort from Mirantis and Rackspace Australia to fix bugs in database migrations in nova. To be frank, we do a poor job of having reliable migrations, even in the relatively simple world of linear migrations. I strongly feel we'd do an even worse job if we had non-linear migrations, and I think we need to require that all migrations be sequential as a matter of policy. Perhaps one day when we're better at writing migrations we can vary that, but I don't think we're ready for it yet.

      Alembic

      An example of an existing user of alembic in openstack is neutron, so I took a look at their code to work out what migrations in nova using alembic might look like. First off, here's the work flow for adding a new migration:

      First off, have a read of neutron/db/migration/README. The process involves more tools than nova developers will be used to, its not a simple case of just adding a manually written file to the migrations directory. First off, you need access to the neutron-db-manage tool to write a migration, so setup neutron.

      Just as an aside, the first time I tried to write this blog post I was on an aeroplane, with no network connectivity. Its is frustrating that writing a new database migration requires network connectivity if you don't already have the neutron tools setup in your development environment. Even more annoyingly, you need to have a working neutron configuration in order to be able to add a new migration, which slowed me down a fair bit when I was trying this out. In the end it seems the most expedient way to do this is just to run up a devstack with neutron configured.

      Now we can add a new migration:

        $ neutron-db-manage --config-file /etc/neutron/neutron.conf \
        --config-file /etc/neutron/plugins/ml2/ml2_conf.ini \
        revision -m "funky new database migration" \
        --autogenerate
        No handlers could be found for logger "neutron.common.legacy"
        INFO  [alembic.migration] Context impl MySQLImpl.
        INFO  [alembic.migration] Will assume non-transactional DDL.
        INFO  [alembic.autogenerate] Detected removed table u'arista_provisioned_tenants'
        INFO  [alembic.autogenerate] Detected removed table u'ml2_vxlan_allocations'
        INFO  [alembic.autogenerate] Detected removed table u'cisco_ml2_nexusport_bindings'
        INFO  [alembic.autogenerate] Detected removed table u'ml2_vxlan_endpoints'
        INFO  [alembic.autogenerate] Detected removed table u'arista_provisioned_vms'
        INFO  [alembic.autogenerate] Detected removed table u'ml2_flat_allocations'
        INFO  [alembic.autogenerate] Detected removed table u'routes'
        INFO  [alembic.autogenerate] Detected removed table u'cisco_ml2_credentials'
        INFO  [alembic.autogenerate] Detected removed table u'ml2_gre_allocations'
        INFO  [alembic.autogenerate] Detected removed table u'ml2_vlan_allocations'
        INFO  [alembic.autogenerate] Detected removed table u'servicedefinitions'
        INFO  [alembic.autogenerate] Detected removed table u'servicetypes'
        INFO  [alembic.autogenerate] Detected removed table u'arista_provisioned_nets'
        INFO  [alembic.autogenerate] Detected removed table u'ml2_gre_endpoints'
          Generating /home/mikal/src/openstack/neutron/neutron/db/migration/alembic_migrations/
        versions/297033515e04_funky_new_database_m.py...done
        


        This command has allocated us a migration id, in this case 297033515e04. Interestingly, the template migration drops all of the tables for the ml2 driver, which is a pretty interesting choice of default.

        There are a bunch of interesting headers in the migration python file which you need to know about:

          """funky new database migration
          
          Revision ID: 297033515e04
          Revises: havana
          Create Date: 2013-11-04 17:12:31.692133
          
          """
          
          # revision identifiers, used by Alembic.
          revision = '297033515e04'
          down_revision = 'havana'
          
          # Change to ['*'] if this migration applies to all plugins
          
          migration_for_plugins = [
              'neutron.plugins.ml2.plugin.Ml2Plugin'
          ]
          


          The developer README then says that you can check your migration is linear with this command:

            $ neutron-db-manage --config-file /etc/neutron/neutron.conf \
            --config-file /etc/neutron/plugins/ml2/ml2_conf.ini check_migration
            


            In my case it is fine because I'm awesome. However, it is also a little worrying that you need a tool to hold your hand to verify this because its too hard to read through the migrations to verify it yourself.

            So how does alembic go with addressing the concerns we have with the nova database migrations? Well, alembic is currently supported by an upstream other than OpenStack developers, so alembic addresses that concern. I should also say that alembic is obviously already in use by other OpenStack projects, so I think it would be a big ask to move to something other than alembic.

            Alembic does allow linear migrations as well, but its not enforced by the tool itself (in other words, non-linear migrations are supported by the tooling). That means there's another layer of checking required by developers in order to maintain a linear migration stream, and I worry that will introduce another area in which we can make errors and accidentally end up with non-linear migrations. In fact, in the example of multiple patches competing to be the next one in the line alembic is worse, because the headers in the migration file would need to be updated to ensure that linear migrations are maintained.

            Conclusion

            I'm still not convinced alembic is a good choice for nova, but I look forward to a lively discussion at the design summit about this.

            Tags for this post: openstack icehouse migrate alembic db migrations
            Related posts: Exploring a single database migration; On Continuous Integration testing for Nova DB

            Comment

            Syndicated 2013-11-03 22:52:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

On Continuous Integration testing for Nova DB

To quote Homer Simpson: "All my life I've had one dream, to achieve my many goals.".

One of my more recent goals is a desire to have real continuous integration testing for database migrations in Nova. You see, at the moment, database migrations can easily make upgrades painful for deployers, normally by taking a very long time to run. This is partially because we test on trivial datasets on our laptops, but it is also because it is hard to predict the scale of the various dimensions in the database -- for example: perhaps one deployment has lots of instances; whilst another might have a smaller number of instances but a very large number of IP addresses.

The team I work with at Rackspace Australia has therefore been cooking up a scheme to try and fix this. For example, Josh Hesketh has been working on what we call Turbo Hipster, which he has blogged about. We've started off with a prototype to prove we can get meaningful testing results, which is running now.

Since we finished the prototype we've been working on a real implementation, which is known as Turbo Hipster. I know it's an odd name, but we couldn't decide what to call it, so we just took a suggestion from the github project namer. Its just an added advantage that the OpenStack Infra team think that the name is poking fun at them. Turbo Hipster reads the gerrit event stream, and then uses our own zuul to run tests and report results to gerrit. We need our own zuul because we want to able to offer federated testing later, and it isn't fair to expect the Infra team to manage that for us. There's nothing special about the tests we're running; our zuul is capable of running other tests if people are interested in adding more, although we'd have to talk about if it makes more sense for you to just run your own zuul.

Generally I keep an eye on the reports and let developers know when there are problems with their patchset. I don't want to link to where the reports live just yet. Right now, there are some problems which stop me from putting our prototype in a public place, though. Consider a migration that takes some form of confidential data out of the database and just logs it. Sure, we'd pick this up in code review, but by then we might have published test logs with confidential information. This is especially true because we want to be able to run tests against real production databases, both ones donated to run on our test infrastructure and ones where a federated worker is running somewhere else.

We have therefore started work on a database anonymization tool, which we named Fuzzy Happiness (see earlier comment about us being bad at naming things). This tool takes markup in the sqlalchemy models file and uses that to decide what values to anonymize (and how). Fuzzy Happiness is what prompted me to write this blog post: Nova reviewers are about to see a patch with strange markup in it, and I wanted something to point at to explain what we're trying to do.

Once we have anonymization working there is one last piece we need, which is database scaling. Perhaps the entire size of your database gives away things you don't want leaked into gerrit. This tool is tentatively codenamed Elastic Duckface, and we'll tell you more about it just as soon as we've written it.

I'd be very interested in comments on any of this work, so please do reach out if you have thoughts.

Tags for this post: openstack turbo_hipster fuzzy_happiness db ci anonymization
Related posts: Nova database continuous integration

Comment

Syndicated 2013-11-02 13:10:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

Starship Troopers (again)




ISBN: 0441783589
Ace (1987), Edition: Reissue, Paperback, 272 pages
LibraryThing
I last read this book almost exactly four years ago. Its still a good read, and I didn't find it as ranty as last time. I do think this is a better story than the movie, as it has more depth. Overall a good read, if not a particularly deep one.

Tags for this post: book robert_a_heinlein combat hugo award combat_suit npr_top_100_sf
Related posts: The Forever War; Starship Troopers; Caves of Steel; Ender's Game; The Diamond Age ; Rendezvous With Rama


Comment

Syndicated 2013-09-30 04:58:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

Call for presentations for the linux.conf.au 2014 OpenStack mini-conference

I've just emailed this out to the relevant lists, but I figured it can't hurt to post it here as well...

linux.conf.au will be hosting the second OpenStack mini-conference to run in Australia. The first one was well attended, and this mini-conference will be the first OpenStack conference to be held on Australia's west coast. The mini-conference is a day long event focusing on OpenStack development and operations, and is available to attendees of linux.conf.au.

The mini-conference is therefore calling for proposals for content. Speakers at the mini-conference must be registered for linux.conf.au 2014 as delegates, or discuss their needs with the mini-conference organizers if that isn't possible.

Some examples of talks we're interested in are: talks from OpenStack developers about what features they are working on for IceHouse; talks from deployers of OpenStack about their experiences and how others can learn from them; talks covering the functionality of OpenStack and how it can be used in new and interesting ways.

Some important details:

  • linux.conf.au runs from 6 to 10 January 2014 in Perth, Australia at the University of Western Australia
  • the mini-conference will be on Tuesday the 7th of January
  • proposals are due to the mini-conference organizer no later than 1 November
  • there are two types of talks -- full length (45 minutes) and half length (20 minutes)


CFP submissions are made by completing this online form: CFP submission form

If you have questions about this call for presentations, please contact Michael Still at openstack-lca2014@lists.stillhq.com for more details.

Tags for this post: conference lca2014 openstack mini-conference rackspace
Related posts: Faster pip installs; Merged in Havana: fixed ip listing for single hosts; Merged in Havana: configurable iptables drop actions in nova; Michael's surprisingly unreliable predictions for the Havana Nova release; Some quick operational notes for users of loop and nbd devices; Exploring a single database migration

Comment

Syndicated 2013-09-04 18:55:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

Exploring a single database migration

Yesterday I was having some troubles with a database migration download step, and a Joshua Hesketh suggested I step through the migrations one at a time and see what they were doing to my sqlite test database. That's a great idea, but it wasn't immediately obvious to me how to do it. Now that I've figured out the steps required, I thought I'd document them here.

First off we need a test environment. I'm hacking on nova at the moment, and tend to build throw away test environments in the cloud because its cheap and easy. So, I created a new Ubuntu 12.04 server instance in Rackspace's Sydney data center, and then configured it like this:

    $ sudo apt-get update
    $ sudo apt-get install -y git python-pip git-review libxml2-dev libxml2-utils
    libxslt-dev libmysqlclient-dev pep8 postgresql-server-dev-9.1 python2.7-dev
    python-coverage python-netaddr python-mysqldb python-git virtualenvwrapper
    python-numpy virtualenvwrapper sqlite3
    $ source /etc/bash_completion.d/virtualenvwrapper
    $ mkvirtualenv migrate_204
    $ toggleglobalsitepackages
    

    Simple! I should note here that we probably don't need the virtualenv because this machine is disposable, but its still a good habit to be in. Now I need to fetch the code I am testing. In this case its from my personal fork of nova, and the git location to fetch will obviously change for other people:

Nova database continuous integration

I've had some opportunity recently to spend a little quality time off line, and I spent some of that time working on a side project I've wanted to do for a while -- continuous integration testing of nova database migrations. Now, the code isn't perfect at the moment, but I think its an interesting direction to take and I will keep pursuing it.

One of the problems nova developers have is that we don't have a good way of determining whether a database migration will be painful for deployers. We can eyeball code reviews, but whether code looks reasonable or not, its still hard to predict how it will perform on real data. Continuous integration is the obvious solution -- if we could test patch sets on real databases as part of the code review process, then reviewers would have more data about whether to approve a patch set or not. So I did that.

At the moment the CI implementation I've built isn't posting to code reviews, but that's because I want to be confident that the information it gathers is accurate before wasting other reviewers' time. You can see results at openstack.stillhq.com/ci. For now, I am keeping an eye on the test results and posting manually to reviews when an error is found -- that has happened twice so far.

The CI tests work by restoring a MySQL database to a known good state, upgrading that database from Folsom to Grizzly (if needed). It then runs the upgrades already committed to trunk, and then the proposed patch set. Timings for each step are reported -- for example with my biggest test database the upgrade from Folsom to Grizzly takes between about 7 and 9 minutes to run, which isn't too bad. You can see an example log at here.

I'd be interested in know if anyone else has sample databases they'd like to see checks run against. If so, reach out to me and we can make it happen.

Tags for this post: openstack rackspace database ci mysql
Related posts: MythBuntu 8.10 just made me sad; Time to document my PDF testing database; Managing MySQL the Slack Way: How Google Deploys New MySQL Servers; I won a radio shark and headphones!; Conference Wireless not working yet?; Drizzle Developer Day; MySQL scaling: query snipers; Links from Rasmus' PHP talk; Faster pip installs; Off to the MySQL tutorials; Recovering lost MythTV recordings; Links from Rasmus' PHP talk; MySQL Workbench; Off to the MySQL tutorials; Thoughts on the first day of the MySQL user's conference; VTA station for the Santa Clara Convention Center; Merged in Havana: fixed ip listing for single hosts; MySQL Workbench; Estimating the progress of queries on MySQL; Is there any way to access the match text in MySQL rlike selects?; Merged in Havana: configurable iptables drop actions in nova

Comment

Syndicated 2013-07-03 03:30:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

We all know that the LCA2014 CFP is open, right?

I just want to make sure that everyone knows that the LCA2014 call for proposals is open. There are two calls this time around -- a call for proposals and a call for miniconfs. The call for proposals closes on 6 July, so you don't have heaps of time left to submit something.

So, if you're interested in speaking at linux.conf.au 2014, in Perth between 6 and 10 January 2014 you should hit up those CFPs now!

Tags for this post: conference lca2014 cfp
Related posts: LCA 2006: CFP closes today; Got Something to Say? The LCA 2013 CFP Opens Soon!; Call for papers opens soon

Comment

Syndicated 2013-06-17 03:00:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

Merged in Havana: fixed ip listing for single hosts

Nova has supported listing the fixed ips for a single host for a while. Well, except for that time we broke it by removing the database call it used and not noticing. My change to fix that situation has just landed, so this should now work again. To list the fixed ips used on a host, do something like:

    nova-manage fixed list hostname


I will propose a backport to grizzly for this now.

Tags for this post: openstack havana fixed_ip nova rackspace
Related posts: Upgrade problems with the new Fixed IP quota; Merged in Havana: configurable iptables drop actions in nova; Michael's surprisingly unreliable predictions for the Havana Nova release; Havana Nova PTL elections; Faster pip installs; Some quick operational notes for users of loop and nbd devices; Further adventures with base images in OpenStack; OpenStack at linux.conf.au 2013; Moving on; Image handlers (in essex); Openstack compute node cleanup

Comment

Syndicated 2013-04-26 00:56:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

Michael's surprisingly unreliable predictions for the Havana Nova release

I should start out by saying that because OpenStack is an open source project, it is hard to know exactly what will land in Havana -- the developers are volunteers, and sometimes things get in the way of them doing the work they intended. However, these are the notes I wrote up on the high points of the summit for me -- I didn't see all the same sessions as other nova developers, so hopefully others will pitch in with their notes as well.

Scheduler

The scheduler seems to be a point of planned work for a lot of people in this release, with talk about having more scheduling code in the common library, and of adding new filter types. There is definite interest in being able to schedule by methods we don't currently support -- things like rack or PDU diversity, or trying to collocate a tenants machines together. HP is also interested in being able to sell dedicated machines to tenants -- in other words, they would guarantee that only one tenants instances appeared on a machine in return for a fee. At the moment this requires setting up a host aggregate for the tenant.

Feeding additional data into scheduling decisions

There is also interest in being able to feed more scheduling information to the nova-scheduler. For example, ceilometer intends to start collecting monitoring data from nova-compute nodes, and perhaps it might inform nova-scheduler that a machine is running hot or has a degraded RAID array. This might also be the source of PDU or CRAC failure information which might affect scheduling decisions -- these later two examples are interesting because they are information where it doesn't make sense to get it from the compute node, the correct location for this information is a data center wide system, not an individual machine. There is concern about nova-scheduler depending on other systems, so these updates from ceilometer will probably be advisory updates, with nova-scheduler degrading gracefully if they are not present or are stale.

Mothballing

This was almost instantly renamed to "shelving", but "swallow / spew" was also considered. This is a request that Rackspace sees from customers -- basically the ability to stop a virtual machine, but keep the UUID and IP addresses associated with the machine as well as the block device mapping. The proposal is to implement this as a snapshot of the machine, and a new machine state. The local disk files for the instance might get deleted if the resources are needed. This would feel like a reboot of an instance to a user.

This is of interest for workloads like "Black Friday" web servers. You could bring a whole bunch up, configure security groups, load balancers, and the applications on the instances and then shelve the instance. When you need the instance to handle load, you'd then unshelve the instance and once it was booted it would just magically start serving. Expect to see shelves instances be cheaper than a running instance, but not free. This is mostly because IP addresses are scarce. Restarting a shelved instance might take a while if the snapshot has to be fetched to a compute node. If you need a more "instant on" bursting capacity, then just leave instances idling and pay full price.

Deferred instance file delete

This is a nice to have requirement for shelving instances, but it is useful for other things as well. This is the ability to delay the deletion of instance files when an instance is torn down. This might end up being expressed as "keep these files for at least X days, unless you are tight on disk resources". I can see other reasons this would be useful -- for example helping support people rescue data from instances users tore down and now want back. It also defers the disk IO from deleting the files until its absolutely necessary. We could also perhaps detect times when the disks are "relatively idle" and use those to clean up file systems.

DNS in nova-network

Expect to see the current DNS driver removed, as no one uses it as best as we can tell. This will be replaced with a simpler drive in nova-compute and the recommendation that deployers use quantum DNS if possible.

Quantum

There is continued work of making quantum the default networking engine for nova. There are still some missing features, but the list of absolutely blocking features is getting smaller. A lot of discussion centered around how to live upgrade clouds from nova-network to quantum. This is not an easy problem, but smart people are looking at it. The solution might involve moving compute nodes over to quantum, and then live migrating instances over to those compute nodes. However, we currently only support one network driver at a time in nova, so we will need to change some code here.

Long running periodic tasks

There will be a refactor of the periodic task code in nova this release to move periodic tasks which incur a lot of blocking IO into separate processes. These processes will be launched by nova-compute, and not be cron jobs or something like that. Most of the discussion was around how to do this safely (eventlet makes it exciting), which is nice in that it indicates some level of consensus that this is needed. The plan for now is to do this in nova-compute, but leave other nova components for later releases.

Libvirt changes

Libvirt is the compute driver I work on, so it's the only one I want to comment on here. The other drivers are doing interesting things as well, I just don't want to get details wrong by not understanding their efforts.

First off, there should be some work done on better console logging in Havana. At the moment we use an unbounded file on disk. This will hopefully become a Unix domain socket managing a ring buffer of some form. The Unix domain socket leaves the option open of later making this serial console interactive, but that's not an immediate goal.

There was a lot of talk about LXC support, and how we need to support file system attachments as well as block devices. There is also some cleanup that can be done for the LXC support in the libvirt to make the code cleaner, but it is not clear who will work on this.

imagebackend.py will probably get refactored, but in ways that don’t make a big difference to users but make it easier to code against (and therefore more reliable). I'm including it here just because I'm excited about that refactor making this code easier to understand.

There was a lot of talk about live migration and the requirement for ssh between compute nodes. Operators don't love that compute nodes can talk to each other, but expect Havana to include some sort of on demand ssh key management, and a later release to proxy that traffic through something like nova-conductor.

Incremental backups are of interest to deployers as well, but there is concern that glance needs more support for chains of images before we can do that.

Conclusion The summit was fantastic once again, and the Foundation did an awesome job of hosting it. It was however a pretty tiring experience, and I'm sure I got some stuff here wrong, or missed things that others would consider important. It would be cool for other developers to write up summaries of what they saw at the summit as well.

Tags for this post: openstack havana rackspace summit nova summary prediction
Related posts: Merged in Havana: configurable iptables drop actions in nova; Havana Nova PTL elections; Upgrade problems with the new Fixed IP quota; Faster pip installs; Some quick operational notes for users of loop and nbd devices; Further adventures with base images in OpenStack; OpenStack at linux.conf.au 2013; Moving on; Image handlers (in essex); Openstack compute node cleanup

Comment

Syndicated 2013-04-19 23:20:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

929 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!