Open Source at Puppet

Gabriel Nagy

Open Source at Puppet

gabi@puppet.com

`$ whoami`

Linux & FOSS user
at Puppet since May
mostly working on open-source projects

what is this?

overview of our projects
how we benefit from /using|being/ open-source
pros & cons of open-source

puppet-agent

collection of software required for puppet and its dependencies to run
bundles the following:
- runtime - we build our own ruby, curl, openssl, boost, etc.
- puppet - Ruby gem, our main project
- facter - Ruby gem (v2), and C++ binary and libraries (v3)
- ... - a bunch of other dependencies (Ruby, C++)

built, tested, and shipped for 45 OSes/platforms (as of November)

`puppet`

configuration management tool, declarative language
you declare the final state of the environment

$ puppet resource user gabi

user { 'gabi':
  ensure  => 'present',
  comment => 'gabi',
  gid     => 1000,
  groups  => ['wheel', 'input', 'lp', 'sys', 'network', 'docker'],
  home    => '/home/gabi',
  shell   => '/bin/zsh',
  uid     => 1000,
}

puppet translates this into platform-specific instructions

`facter`

provides facts about the system (network, disks, memory, OS)
current version in C++, can be extended with custom Ruby facts
can be used through a shared library, or standalone by calling its executable

$ facter system_uptime --json

{
  "system_uptime": {
    "days": 0,
    "hours": 1,
    "seconds": 4322,
    "uptime": "1:12 hours"
  }
}

how we use open source

keep our runtime up-to-date by patching CVEs
October vulnerabilities:
- Ruby
- OpenSSL
- cURL
all upgrades were problematic

`ruby`

hotfix release did not install
the version we had to upgrade to was 2.4.8

2.4.9 was released a couple of minutes before we checked ruby-lang.org

`openssl`

new version did not compile on Windows

the openssl release prep commit itself did not pass CI
issue was fixed after the release

`curl`

new version did not compile on Solaris
issue was fixed after the release

conclusions?

open-source projects are widely used and understaffed
- curl has been maintained for over 20 years by a single person
- openssl had 2 developers before the Heartbleed vulnerability
- ruby - around 20 people are actively contributing
most of the time projects are maintained through volunteer efforts

if something doesn’t work, don’t immediately try to fix it. find out if someone else has done it already. it’s open-source

how the community helps

had to update our compile/link flags for our vendored libraries and binaries
- easy: check how Debian/CentOS/Arch build our packages and do the same
our latest puppet release had a regression
- community provided a fix shortly after we released
  
  as a newcomer, puppet being open-source helped a lot. you can search the internet for most issues and it’s high chance that someone else might have encountered what you are looking for

how we contribute back

we use ruby extensively on non-standard platforms
sometimes we discover issues…

segfaulting with `facter`

Facter.add('ip') do
  url = 'https://api.ipify.org?format=json'
  # {"ip":"79.114.90.219"}
  response = open(url).read
  if !response.to_s.empty?
    result = JSON.parse(response)
    setcode { result['ip'] }
  end
end

custom fact that queries an API for your public IP

$ facter ip
79.114.90.219

on windows though…

$ facter ip

C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/2.5.0/net/protocol.rb:45: [BUG] Segmentation fault
ruby 2.5.3p105 (2018-10-18 revision 65156) [x64-mingw32]

-- Control frame information -----------------------------------------------
c:0024 p:---- s:0165 e:000164 CFUNC  :wait_readable
c:0023 p:0093 s:0160 e:000159 METHOD C:/puppet/lib/ruby/2.5.0/net/protocol.rb:45
c:0022 p:0557 s:0153 E:001568 METHOD C:/puppet/lib/ruby/2.5.0/net/http.rb:981
c:0021 p:0004 s:0140 e:000139 METHOD C:/puppet/lib/ruby/2.5.0/net/http.rb:920
c:0020 p:0029 s:0136 e:000135 METHOD C:/puppet/lib/ruby/2.5.0/net/http.rb:909
c:0019 p:0521 s:0132 e:000131 METHOD C:/puppet/lib/ruby/2.5.0/open-uri.rb:337
c:0018 p:0017 s:0111 e:000110 METHOD C:/puppet/lib/ruby/2.5.0/open-uri.rb:755
c:0017 p:0029 s:0104 e:000103 BLOCK  C:/puppet/lib/ruby/2.5.0/open-uri.rb:226 [FINISH]
c:0016 p:---- s:0101 e:000100 CFUNC  :catch
c:0015 p:0365 s:0096 E:0010e8 METHOD C:/puppet/lib/ruby/2.5.0/open-uri.rb:224
c:0014 p:0328 s:0081 e:000080 METHOD C:/puppet/lib/ruby/2.5.0/open-uri.rb:165
c:0013 p:0018 s:0069 e:000068 METHOD C:/puppet/lib/ruby/2.5.0/open-uri.rb:735
c:0012 p:0071 s:0063 e:000062 METHOD C:/puppet/lib/ruby/2.5.0/open-uri.rb:35
c:0011 p:0007 s:0055 e:000054 BLOCK  C:/cygwin64/home/Administrator/facts/ip.rb:10
c:0010 p:0030 s:0052 E:000948 BLOCK  C:/puppet/lib/ruby/2.5.0/timeout.rb:93
c:0009 p:0005 s:0046 e:000045 BLOCK  C:/puppet/lib/ruby/2.5.0/timeout.rb:33 [FINISH]
c:0008 p:---- s:0043 e:000042 CFUNC  :catch
c:0007 p:0044 s:0038 e:000037 METHOD C:/puppet/lib/ruby/2.5.0/timeout.rb:33
c:0006 p:0113 s:0032 E:000650 METHOD C:/puppet/lib/ruby/2.5.0/timeout.rb:108
c:0005 p:0021 s:0020 E:000748 BLOCK  C:/cygwin64/home/Administrator/facts/ip.rb:9 [FINISH]
c:0004 p:---- s:0014 e:000013 CFUNC  :instance_eval
c:0003 p:---- s:0011 e:000010 CFUNC  :add
c:0002 p:0034 s:0006 E:000790 TOP    C:/cygwin64/home/Administrator/facts/ip.rb:5 [FINISH]
c:0001 p:0000 s:0003 E:000640 (none) [FINISH]

the elephant in the room

this code is not directly evaluated by the Ruby interpreter
the Facter.add implementation looks like this:

VALUE module::ruby_add(int argc, VALUE* argv, VALUE self)
{
    return safe_eval("Facter.add", [&]() {
	auto const& ruby = api::instance();

	VALUE fact_self = from_self(self)->create_fact(argv[0]);

	...

	return fact_self;
    });
}

what we knew

this worked on ruby 2.4 but crashed on 2.5
only happened on Windows
somehow linked to the URL opening part

next steps

check what changed in ruby between 2.4 and 2.5

$ git diff --stat v2_4_5..v2_5_3
...
6101 files changed, 340476 insertions(+), 79434 deletions(-)

only 60 commits under win32/

$ git rev-list --count v2_4_5..v2_5_3 -- win32/
60

what we could have done

what we did

windows sockets?

a hunk of commit e33b169 - win32.c: vm_exit_handler

this replaces all references of NtSocketsInitialized with 1 before compile time

`NtSocketsInitialized`

this variable is referenced over 30 times in the file, similar to the code below:

if (!NtSocketsInitialized) {
    StartSockets();
}

after preprocessing, the bit becomes:

[~/repo/ruby]$ cpp win32/win32.c

if (!1) {
    StartSockets();
}

StartSockets() is the function responsible to initialize the Winsock DLL
Winsock handles internet I/O requests for Windows
There’s no networking without Winsock

getting to the point

after commit e33b169, StartSockets() goes from possibly being called in 34 places

to getting called once

void rb_w32_sysinit(int *argc, char ***argv)
{
    // ...
    tzset();
    init_env();
    init_stdhandle();
    atexit(exit_handler);

    // Initialize Winsock
    StartSockets();
}

rb_w32_sysinit is a function that we have to call manually if we’re embedding the Ruby interpreter
to be platform-agnostic we can call the main sysinit function:

void ruby_sysinit(int *argc, char ***argv)
{
#if defined(_WIN32)
    rb_w32_sysinit(argc, argv);
#endif
...
}

fixing stuff

before calling ruby_sysinit

#include <ruby.h>

int main()
{
    ruby_init(); // sets up some basic things
    char* options[] = { "-v", "-e", "" };
    ruby_options(3, options); // sets up more stuff

    rb_require("open-uri");
    rb_eval_string("puts open('http://api.ipify.org?format=json').read");
}

C:/tools/ruby26/lib/ruby/2.6.0/net/http.rb:949:in `rescue in block in connect':
Failed to open TCP connection to api.ipify.org:80 (getaddrinfo: Either the application has not called WSAStartup, or WSAStartup failed.) (SocketError)

fixing stuff

after calling ruby_sysinit

#include <ruby.h>

int main()
{
    ruby_init(); // sets up some basic things
    char* options[] = { "-v", "-e", "" };
    ruby_options(3, options); // sets up more stuff

    int sysinit_opts_size = 1;
    char const* sysinit_opts[] = { "ruby" };
    ruby_sysinit(&sysinit_opts_size, (char***)(&sysinit_opts));

    rb_require("open-uri");
    rb_eval_string("puts open('http://api.ipify.org?format=json').read");
}

{"ip":"192.69.65.12"}

finishing up

opened a pull request on GitHub for ruby/ruby

finishing up

merged after almost 2 months

conclusions

the Ruby C API is huge and largely undocumented
the best way to understand things is to go through the code

links & questions

presentation: gabrielnagy.github.io/open-source-presentation
our projects:
ruby:

Gabriel Nagy

Open Source at Puppet

$ whoami

what is this?

puppet-agent

puppet

facter

how we use open source

ruby

openssl

curl

conclusions?

how the community helps

how we contribute back

segfaulting with facter

on windows though…

the elephant in the room

what we knew

next steps

what we could have done

what we could have done

what we did

what we did

windows sockets?

NtSocketsInitialized

getting to the point

fixing stuff

fixing stuff

finishing up

finishing up

conclusions

links & questions

`$ whoami`

`puppet`

`facter`

`ruby`

`openssl`

`curl`

segfaulting with `facter`

`NtSocketsInitialized`