Category Archives: General

Linux 4.10

As always, KernelNewbies is the place to go for a great summary of new kernel features. Here’s some 4.10 highlights I’m interested in.

BPF for lightweight tunnel encapsulation commit, https://git.kernel.org/torvalds/c/f74599f7c5309b21151233b98139e9b723fd1110

 

TCP: sender chronographs instrumentation. This feature exports the sender chronograph stats via the socket SO_TIMESTAMPING channel. Currently it can instrument how long a particular application unit of data was queued in TCP by tracking SOF_TIMESTAMPING_TX_SOFTWARE and SOF_TIMESTAMPING_TX_SCHED. Having these sender chronograph stats exported simultaneously along with these timestamps allow further breaking down the various sender limitation. For example, a video server can tell if a particular chunk of video on a connection takes a long time to deliver because TCP was experiencing small receive window commit, commit, commit, commit, commit, commit

 

sched/act_mirred: Implement the corresponding ingress actions TCA_INGRESS_REDIR and TCA_INGRESS_MIRROR (Up until now, action mirred supported only egress actions (either TCA_EGRESS_REDIR or TCA_EGRESS_MIRROR). This allows attaching filters whose target is to hand matching skbs into the rx processing of a specified device commit, commit, commit, commit

 

Add the LRU versions of the existing BPF_MAP_TYPE_HASH and BPF_MAP_TYPE_PERCPU_HASH maps: BPF_MAP_TYPE_LRU_HASH and BPF_MAP_TYPE_LRU_PERCPU_HASH sample, commit, commit, commit, commit, commit

 

Since the dawn of time, the way Linux synchronizes to disk the data written to memory by processes (aka. background writeback) has sucked. When Linux writes all that data in the background, it should have little impact on foreground activity. That’s the definition of background activity…But for a long as it can be remembered, heavy buffered writers have not behaved like that. For instance, if you do something like $ dd if=/dev/zero of=foo bs=1M count=10k, or try to copy files to USB storage, and then try and start a browser or any other large app, it basically won’t start before the buffered writeback is done, and your desktop, or command shell, feels unreponsive. These problems happen because heavy writes -the kind of write activity caused by the background writeback- fill up the block layer, and other IO requests have to wait a lot to be attended (for more details, see the LWN article).

This release adds a mechanism that throttles back buffered writeback, which makes more difficult for heavy writers to monopolize the IO requests queue, and thus provides a smoother experience in Linux desktops and shells than what people was used to. The algorithm for when to throttle can monitor the latencies of requests, and shrinks or grows the request queue depth accordingly, which means that it’s auto-tunable, and generally, a user would not have to touch the settings. This feature needs to be enabled explicitly in the configuration (and, as it should be expected, there can be regressions)

Recommended LWN article: Toward less-annoying background writeback

eBPF Map Size Limits

Recently, I’ve done some work with eBPF and specifically the in-kernel maps that are manipulated and shared by both kernel and user space code.

When doing this I ran into permission errors when installing large maps. It took a little while to figure out that the cause of this was the root user’s locked memory limit being too low (thanks Daniel Borkmann).

The locked memory limit is modified with ulimit:

ulimit -l unlimited

Processing the U.S Election

Like a lot of people, I’m still trying to process the U.S. presidential election results and understand what it means for the future. Writing this is part of that process.

Before delving into the results and how they were achieved, I want to make it clear that I’m not a big fan of Hillary Clinton. While she certainly represented less risk than Donald Trump, that reduced risk comes from more of the same which clearly isn’t working well for a part of the U.S. population. I’m quite sympathetic to the idea that a non-politician or Washington outsider is required to change how things are done. Read Republic Lost for some insight into how things now are very broken.

Additionally, I very strongly dislike the trend towards recurring political families. Hereditary privilege is a cancer in all forms – the world moved forward when we removed ruling families. This trend also smacks of cronyism and the power of brand over substance. In Canada, we’ve seen this trend with the election of Justin Trudeau, the son of a Prime Minister from 32 years ago. In the last federal election, I very seriously considered not voting for the Liberal Party despite my general support of their policies just because of the stink of a new class of ‘ruling families’.

When reading the rest of this, please understand that it doesn’t come from a place of being angry or sad that Hillary Clinton lost.

It seems that there were three high level concerns, desires or ideas that drove the election’s outcome:

  1. A general dislike for Hillary Clinton. I don’t know how much of this is sexism or the result of years of anti-Clinton messaging. I’m not going to say anything else about this.
  2. Enormous, and legitimate, dissatisfaction from the middle to low-end of the economic spectrum. Globalism and technological change have left these people behind.
  3. Racism. Both real and as a scapegoat for #2.

I don’t know the ratio to which these three ideas affected the outcome. 33-33-33? 10-45-45? To whatever extent the reason was #1, it is not a long term problem. Hillary Clinton won’t run next time. #2 and #3 are real, structural problems that present a risk to the future.

It is very clear that some part of the Trump vote was driven by pure racism. It’s hardly a representative sample but it is unreasonable to say that racism played no role given some of the incidents that have happened in the last few days:

It’s impossible to rationalize with or about this kind of hate and irrational behavior. I’m not going to even try other than to say it makes me sad and angry that there are still human beings alive in 2016 that feel this type of behavior is acceptable.

What scares me, is the interplay between #2 and #3.

I do believe there is a legitimate case to make that the low-skilled and uneducated (not lazy, not unintelligent) worker has a right to be angry. Globalism has moved jobs to other countries with lower employment standards and technology has contributed to fewer low skilled jobs even when the manufacturing remains in rich countries. Canada faces a very similar problem. In Ontario, the country’s equivalent of the rust belt, hundreds of thousands of good paying manufacturing jobs have disappeared. The small town and city I grew up near have experienced this and I fear that they are, in part, still surviving on the wages earned and homes bought many years ago. [In these particular cases, the in-flux of retired farmers who were able to do very well on increased land values has also reduced the effects of fewer manufacturing jobs] Despite the loss in high paying manufacturing jobs, it does seem that the effect on the population in Ontario has been less severe than in the U.S. – I’ll get back to that later.

Let’s assume for a minute that racism played no role in Trump winning the election and that these disaffected workers voted to get good jobs back. That, indeed, is a big part of what Trump focused on and promised to do. He promised to accomplish this by renegotiating NAFTA, by labeling China a currency manipulator and by applying tariffs to imported products.

The problem is that none of this will work – certainly not in a time frame that will matter to his presidency.

Even if these measures are successful in motivating companies to move production back to the U.S. these types of changes take years, maybe decades. Think through the time line:

  • Policy and law changes such as tariffs or renegotiating NAFTA make it more profitable to build products in the U.S. At best, these changes make it into law in one year.
  • Once these changes are in effect, companies start seeing changes to their bottom line and this motivates thinking about changing where they build products. It will take a year for this to really hit most companies (yearly results). Some companies may feel the impact in a quarter or two. So realistically, most companies won’t start thinking about making changes until 6 months to a year after the impact starts.
  • Presumably, the changes make it more profitable to manufacture in the U.S. Now the company needs to balance the loss in profit against the cost of building new manufacturing plants. To simplify, let’s say the law changes make it $1000 cheaper to build a car in the U.S. and the cost to build a new assembly plant is $1B. This means 1M cars can be built at the existing plant before the bottom line impact makes it a good business decision to build a new plant. Let’s call this 2 years.
  • Once the decision is made to build a new plant, it takes years to plan and build a large manufacturing plant. Free land has to be located and a new supply chain created. 5 years?

Add that up – it’s nearly a decade. Just like it took decades for manufacturing to move out of the U.S., for practical reasons, it will take decades for it to move back even if the financial incentives are in-place.

The march of technology must also be considered. Modern manufacturing takes much few employees than it used to. Any newly created U.S. manufacturing capacity will need fewer people than the plants currently operating in other countries because these new plants will have the latest technology. In a perfect world where the incentives align to drive all the lost manufacturing back to the U.S., far fewer people will get good jobs out of this than the jobs that have been lost.

And the loss of jobs for low skilled workers is only going to get worse. Consider the effect that just one new technology, self driving vehicles is going to have. There are roughly 3.5 million professional truck drivers and 233K taxi cab drivers in the U.S. As for profit businesses, I believe these sectors are going to aggressively switch to self-driving vehicles as soon as the business case makes sense. To do not do so will put the laggards out of business. Additionally, a transportation company doesn’t have a sentimental attachment or ‘like’ driving the way consumers do.

So, policy changes cannot drive manufacturing jobs back to the U.S. in a time horizon that benefits an 8-year Trump presidency. And what manufacturing is incentivized back to the U.S. will employ far fewer people than it did when it left. Technological advancements are about to put new groups of workers into the same situation as workers from the manufacturing sector. Finally, new manufacturing capacity is unlikely to be built in the northern areas where it previously existed because the plants that moved to the southern U.S. already pay much less than the northern, unionized plants they replaced.

In summary, there is no chance of Trump keeping his promise to return good, high paying jobs to the justifiably angry people he committed to.

This is where I get scared and it involves the interplay with #3 – Racism.

When Trump fails to deliver relief to the people who cried out for change by voting for him, these people are going to be angry. The confidence in the institutions that hold society together will be further undermined and someone, something or some group will be the scapegoat. I don’t know if Trump himself will blame ‘the liberals’, or foreign governments, or the Republican establishment that loves trade, or the immigrants – but someone will. And given that racism was used by Trump to stir up the support that won him the presidency, I believe there is a good chance it will also be the driving factor in which group gets will be the scapegoat. Fear of the ‘other’ is something that just comes to our species too easily.

What could bring relief to the people affected negatively by globalism and technological change? Sadly, it’s pretty much the opposite of what Trump has promised and the affected people bought into.

Trump has promised huge tax cuts and changes to school systems. Let’s take each of those in turn. Less tax revenue means fewer resources for the government to effectively help these people. This will reduce what little social safety net the U.S. does have and put people in even more desperate situations. On the school front, allowing ‘choice’ sounds like a common sense thing to do but the only people who have the ability to make choices on where their children go to school are the reasonably well off. Choice in this context means more movement of wealthier students to better schools which will further reduce the attention, quality and funding for the children who need it the most to get out of the hole their parent’s find themselves in for no fault of their own.

I mentioned earlier that Ontario, Canada’s rust belt equivalent, has also experienced a decimation of manufacturing jobs. However, the impact here has been much less dramatic and I think it points in the direction of a solution to the problems at hand [1].

Canada has a free public health system. Manufacturing workers who have lost their jobs aren’t watching their friends and families lose their homes to high medical bills or worrying that it could be them next. They may not be happy that they aren’t making high union wages anymore but that’s a far better situation than being on the street.

Secondly, Ontario has a great K-12 public school system (#4 on that report vs. U.S. at 23) and great Universities which are much more affordable than their U.S. equivalents. Yes, it’s expensive to run a strong public school system. Ontario’s teachers are among the highest paid in the OECD. As a taxpayer who could easily afford to send my children to private school if I wanted to, I have no problems paying for this level of public education. In general, I believe private schools to be a net negative on society. Taking the wealthy people who have the time and money to influence politics out of the public school system results in less time and money going into the public school system. It’s a death spiral.

Public health care, good public education and other social safety nets mean that not all hope is lost for manufacturing workers who have lost their high paying jobs. If they weren’t over extended, lower paying service jobs can continue to pay the mortgage and losing health insurance isn’t a concern either. At the same time, they don’t feel like their children are stuck in the same situation. Their children can get a good K-12 education and even a good college or university education on a much lower family income.

Believing that your children will have it better than you is a powerful motivator. It’s enough that while the affected people are not happy about no longer being able to afford a cottage or boat, they aren’t crying for a revolution or looking for scapegoats either.

In summary, the U.S. now has:

This is a slow burning wick on the way to a powder keg. It’s scary.

[1] – Please don’t read this as a Canada vs. U.S. thing. We have lots of our own problems but it’s insane not to look around the world to see how other countries are dealing with this problem.