About

Feeds

 

〰 Tidal, Archiloque's feed reader

Hacker NewsShow HN: Paper Programs – run JavaScript on pieces of paper
Comments
Hacker NewsThe ‘Frequent Flier’ Program That Grounded a Hospital’s Soaring Costs
Comments
Hacker NewsGoogle thinks the sun is 15.81 light years from earth
Comments
Hacker NewsVergeSense (YC S17) Is Hiring Systems Test Engineers in SF
Comments
Hacker NewsNursing Home Recreates Communist East Germany for Dementia Patients
Comments
Les Crises[Vidéo] Financement du Terrorisme (1/4) : Richard Labévière

Grace à votre soutien, nous avons pu filmer l’important Colloque sur le Financement du Terrorisme organisé par le CF2R, le 5 décembre 2017, que je vous recommande particulièrement au vu de la qualité des intervenants.

Richard Labévière introduit ici la problématique :

Source : Youtube, 05-12-2017

La suite dans les prochains jours…

Lire la suite

QC RSSThe Robot Birds And The Virtual Bees
Les CrisesLes élites « n’ont plus de crédibilité » : une interview du journaliste Chris Hedges, par David North

Source : World Socialist Web Site, David North, 06-10-2017

Lundi, le président du Comité de rédaction international du WSWS [World Socialist Web Site, organe de nouvelles et d’analyses du Comité international de la Quatrième Internationale, NdT] David North, a interviewé Chris Hedges, journaliste, auteur, conférencier et ancien correspondant du New York Times, lauréat du Prix Pulitzer. Parmi les livres les plus connus de Hedges, War is a Force That Gives Us Meaning, The Death of the Liberal Class, Empire of Illusion : the End of Literacy and the Triumph of Spectacle, Days of Destruction, Days of Revolt, qu’il a coécrit avec le dessinateur Joe Sacco, et Wages of Rebellion : Moral Imperative of Revolt.

Chris Hedges

Dans un article paru dans Truthdig le 17 septembre, Hedges fait référence à la couverture par le WSWS de la censure de Google envers les sites de gauche et met en garde contre la prolifération de « listes noires, de censure et de diffamation des dissidents en tant qu’agents étrangers de la Russie et pourvoyeurs de ‘fake news’ ».

Hedges a écrit que « le ministère de la Justice a appelé RT America et ses ‘associés’ – ce qui peut signifier des gens comme moi – à s’inscrire en vertu de la Foreign Agent Registration Act. Il ne fait aucun doute que l’État ploutocratique sait que la plupart d’entre nous ne voudrons pas s’enregistrer en tant qu’agents étrangers, ce qui signifie que nous serons bannis des ondes. Ceci, je le soupçonne, est l’intention. »

Lire la suite

Hacker NewsFacebook open sources Detectron
Comments
Hacker NewsThe unlikely success of California’s Sonic.net (2012)
Comments
Hacker NewsWhy Discrete Math Is Important
Comments
Game WisdomDissecting Design: Dawn of Discovery

Dissecting Design: Dawn of Discovery Josh Bycer josh@game-wisdom.com

This week on Dissecting Design, I took a look at Dawn of Discovery, AKA Anno 1404, to talk about its take on city building and strategy gameplay.

 

The post Dissecting Design: Dawn of Discovery appeared first on Game Wisdom.

Hacker NewsRunning BBS Door Games on Windows 10 with GameSrv and DOSBox
Comments
Hacker NewsExploring permutations and a mystery with BSD and GNU split filenames
Comments
Hacker NewsBad month for the main thread
Comments
Hacker NewsNaomi Parker Fraley, the Real Rosie the Riveter, Dies at 96
Comments
Hacker NewsExploring handwritten digit classification: a tidy analysis of the MNIST dataset
Comments
Hacker NewsAtrium is hiring ML / NLP experts to `[[-1, 0], [0, -1]]` the legal industry*
Comments
Hacker NewsTiny, Wealthy Qatar Goes Its Own Way, and Pays for It
Comments
Hacker NewsThe 10,000 Year Clock
Comments
Hacker NewsTesla ‘on Autopilot’ slams into parked fire truck on freeway
Comments
Hacker NewsDelta Air Lines Tightens Rules for Service and Support Animals
Comments
Hacker NewsIn-N-Out managers are earning a lot more than you think
Comments
Ars TechnicaNet neutrality is bad? 1 million PornHub employees can’t be wrong. Oh, wait.

Enlarge / FCC Chairman Ajit Pai with his oversized coffee mug in November 2017. (credit: Getty Images | Bloomberg)

WASHINGTON, DC—If Federal Communications Commission Chairman Ajit Pai actually allowed the weight of public comments on the FCC's proposed changes to network neutrality regulations to sway (or confirm) his position, he seems to have given more credence to the "opinions" of spam-generating software "bots" than actual citizens, researchers have found.

At the Shmoocon information security conference on Saturday, Leah Figueroa, lead data engineer at the data analytics software company Gravwell, presented a detailed analysis of the public comments submitted to the FCC regarding network neutrality. Applying filters to the over 22 million comments submitted to the FCC, Figueroa and her team attempted to identify which comments were submitted by real US citizens—and which were generated by bulk-uploading bots

At the end of September, Figueroa said, she and her team pulled in all of the submitted comments from the FCC site and applied a series of analytical steps to separate "organic" comments—those most likely to have been submitted by actual human beings—from comments submitted by automated systems ("bots") using faked personal data.

Read 7 remaining paragraphs | Comments

Ars TechnicaVideo demonstrates the marvel of CRT displays at 380,000 frames per second

Enlarge (credit: The Slow Mo Guys)

We spend a lot of time reading about the differences between display technologies like LCD and OLED, which, like all display technologies, are built to fool our eyes into seeing things that are only simulated, not real, like colors, or realistic movement. But it helps to see it in action.

A video from YouTube channel The Slow Mo Guys (originally reported on by Motherboard) vividly illustrates how CRT, LCD, and OLED displays work by either zooming in very close or by recording in insane frame rates at ultra slow motion.

You'll still find enthusiasts who insist that it's all been downhill since CRT monitors and TVs went sunset for most of the market. While this video doesn't make much of a case for CRT's relative quality, it does show that they were engineering marvels for their time.

Read 4 remaining paragraphs | Comments

Hacker NewsUniversities Are Becoming Hedge Funds with Schools Attached (2016)
Comments
Hacker NewsOne data scientist on the hype around artificial intelligence (2017)
Comments
Hacker NewsKeep Your Hands on the Wheel and Don't Look Down
Comments
Ars TechnicaTrump puts 30% tariff on imported solar cells and modules

Enlarge / Long Island solar farm. (credit: Brookhaven National Lab)

On Monday afternoon, the Trump administration released a fact sheet (PDF) detailing new tariffs on imports, including a tariff schedule for solar cells and modules starting at 30 percent.

The solar tariff determination had been tensely anticipated by the US solar industry, with manufacturers arguing that cheap imports from Asia have harmed their businesses. Solar installers, financiers, and sales people, however, argue that cheap imports have created a bigger boom in employment than manufacturing ever could.

The news is likely a blow to the wider solar industry, although it's not entirely unexpected. Trump has been vocal about his preference for tariffs and has shown little desire to extend a hand out to the solar industry, which is often seen as a competitor with fossil fuels. When the International Trade Commission (ITC) voted in favor of imposing tariffs on solar imports in September, the trade association Solar Energy Industries of America (SEIA) prepared for the worst.

Read 10 remaining paragraphs | Comments

Ars TechnicaMaking tools gives crows a big food boost

Enlarge / A crow gets to work manufacturing a tool. (credit: Jolyon Troscianko)

Tool use among animals isn't common, but it is spread widely across our evolutionary tree. Critters from sea otters to cephalopods have been observed using tools in the wild. In most of these instances, however, the animal is simply using something that's found in its environment, rather than crafting a tool specifically for a task. Tool crafting has mostly been seen among primates.

Mostly, but not entirely. One major exception is the New Caledonian crow. To extract food from holes and crevices, these birds use twigs or stems that are found in their environment without modification. In other environments, however, they'll remove branches from plants and carefully strip parts of the plant to leave behind a hooked stick. The behavior takes over a minute, and the crows will typically carry the tool with them when they explore new sites, and they will sometimes store it for future use.

Understanding how this complex behavior came about in crows requires us to understand the evolutionary advantages that might be had from a good tool. A group of researchers, mostly from the University of St. Andrews, has now done just that: the researchers have quantified how tool manufacture influences food harvesting. The results show that the use of bird-crafted tools can increase food extraction by up to 12 times the rate the crows could achieve by using unmodified sticks.

Read 5 remaining paragraphs | Comments

Hacker NewsHow US 'get out of jail free' cards work
Comments
Hacker NewsOnce seniors are too old to drive, our transportation system fails them
Comments
Hacker NewsElixir on Google Cloud Platform and App Engine
Comments
jwzShe's a Flight Risk is counting down to something

Previously, previously, previously, previously, previously.

Hacker NewsCheckr (YC S14) is hiring engineers to build the future of online trust
Comments
Hacker NewsNetflix is now worth more than $100B
Comments
Ars TechnicaSpaceX gets good news from the Air Force on the Zuma mission

Enlarge / The launch of Zuma was pretty, but the aftermath has been anything but. (credit: SpaceX)

A little more than two weeks have passed since the apparent loss of the highly classified Zuma mission. Since then, SpaceX has publicly and privately stated that its Falcon 9 rocket performed nominally throughout the flight—with both its first and second stages firing as anticipated.

Now, the US Air Force seems to be backing the rocket company up. "Based on the data available, our team did not identify any information that would change SpaceX's Falcon 9 certification status," Lieutenant General John Thompson, commander of the Space and Missile Systems Center, told Bloomberg News. This qualified conclusion came after a preliminary review of data from the Zuma launch. That's according to Thompson, who said the Air Force will continue to review data from all launches.

However tentative, this statement buttresses the efforts by SpaceX to say that, from its perspective, the mission was a success. The statement also adds to the concerns of Northrop Grumman, which built the Zuma payload and the adapter that connected it to the Falcon 9 rocket. Northrop Grumman was also responsible for separating after the second stage of the Zuma rocket reached space. The aerospace veteran has yet to publicly comment on specifics of the Zuma mission since the launch.

Read 2 remaining paragraphs | Comments

Hacker NewsTrump imposes 30% tarriff on solar panel imports
Comments
Hacker NewsU.S. Imposes Trade Tariffs, Signaling Tougher Line on China
Comments
Hacker NewsNetflix adds 8.3M subscribers in Q4, revenue up 33%
Comments
Hacker NewsChina, Unhampered by Fear, Races Ahead in Gene-Editing Trials
Comments
Hacker NewsTrump Taxes Solar Imports in Biggest Blow to Renewables Yet
Comments
Hacker NewsThe U.S. will impose duties on solar equipment and washing machines made abroad
Comments
Hacker NewsIsomorphic Go
Comments
Hacker NewsWaymo heads to Atlanta to test its self-driving cars
Comments
Ars TechnicaSorry, FCC: Montana is enforcing net neutrality with new executive order

Enlarge / Montana Governor Steve Bullock. (credit: Getty Images | William Campbell )

Montana will require Internet service providers to follow net neutrality principles in order to receive state government contracts.

Governor Steve Bullock, a Democrat, today signed an executive order imposing that requirement beginning July 1, 2018.

"There has been a lot of talk around the country about how to respond to the recent decision by Federal Communications Commission to repeal net neutrality rules, which keep the Internet free and open," Bullock said. "It's time to actually do something about it. This is a simple step states can take to preserve and protect net neutrality. We can't wait for folks in Washington, DC, to come to their senses and reinstate these rules."

Read 10 remaining paragraphs | Comments

Hacker NewsCoinbase booked $1B in revenue last year – told VCs to back off
Comments
Hacker NewsNetflix plans to spend $8B to make its library 50 percent original by 2018
Comments
Ars TechnicaWindows VR headsets now available with deep discounts

Enlarge / An array of Windows Mixed Reality headsets. (credit: Microsoft)

With its Windows VR headsets, Microsoft wanted to make it simpler and cheaper to get into PC-based virtual reality.

But perhaps not quite this cheap: most of the Windows VR headsets on the market are now available on Amazon in the US for around 50 percent off: for as little as $200, you can get a headset complete with a pair of motion controllers that'll run Windows Mixed Reality software and which has beta quality support for SteamVR titles too.

When first announced, Microsoft promised its headsets would be around $300-500, compared with the $600 or more for the Oculus Rift and HTC Vive. Since then, both the Rift and the Vive have some big price cuts of their own, and while the Windows VR devices do still retain the pricing edge, the difference is much less pronounced than it once was. For the moment, the Windows hardware retains one advantage—it doesn't need base stations to track movement because all the tracking is handled in the headset itself, which makes installation and setup substantially easier. But this benefit, too, is set to disappear in the near term, as this style of tracking is going to become the norm.

Read 1 remaining paragraphs | Comments

Hacker NewsFacebook invents new unit of time – flicks – slightly larger than a nanosecond
Comments
Hacker News“Get Out of Jail Free” Cards
Comments
Hacker NewsPrivate landowners blocking access to public lands in the American West
Comments
Hacker NewsOne Bitcoin miner is buying 20,000 16nm wafers from TSMC per month
Comments
Hacker NewsAll Blizzard games vulnerable to DNS rebinding attack
Comments
Ars TechnicaA Comcast net neutrality commitment from the NBC merger just expired

(credit: https://arstechnica.com/author/aurich-lawson/)

A commitment made by Comcast to follow net neutrality rules expired on Saturday, seven years after the cable company agreed to the requirements in order to purchase NBCUniversal.

When Comcast bought NBC in 2011, it pledged to follow the net neutrality rules the Federal Communications Commission had passed in 2010 even if those rules were later overturned in court. Comcast thus continued to face rules against blocking, throttling, and paid prioritization even after a federal appeals court struck down that version of net neutrality rules in January 2014. Comcast (but not other ISPs) faced net neutrality requirements for more than a year until June 2015, which is when a new set of net neutrality rules took effect.

But Comcast's merger agreement with the FCC expired, as per schedule, on January 20. The expiration, combined with the FCC's decision last month to repeal the industry-wide net neutrality rules implemented in 2015, will free Comcast of FCC oversight when it comes to net neutrality. Comcast will still face some merger-related oversight from the Department of Justice until September, though.

Read 23 remaining paragraphs | Comments

Hacker NewsFossil Discoveries Challenge Ideas About Earth’s Start
Comments
Hacker NewsExperiences with running PostgreSQL on Kubernetes
Comments
Hacker NewsMontana becomes first state to implement net neutrality after FCC repeal
Comments
Hacker NewsArtist Nan Goldin​ takes on the billionaire family behind OxyContin
Comments
Hacker NewsCastle Game Engine 6.4 released – physics, iOS services, shader pipeline upgrade
Comments
Hacker NewsGetting High with Benjamin and Burroughs (2008)
Comments
Hacker NewsThat Time a U.S. Bomber Crashed on Greenland with 4 Nukes Aboard
Comments
Ars TechnicaAn end to in-flight Wi-Fi misery is at hand with Gogo’s 2Ku

Enlarge / You'll know your plane has 2Ku if you see a larger radome on it, like the one pictured above. (credit: Gogo Air)

For this demo, Gogo Air provided a round-trip ticket on Delta Air Lines from DCA>DTW>DCA. I sat in coach and never left Detroit airport before boarding the return leg.

If you're one of those people with the misfortune to follow me on Twitter, you might have noticed the occasional complaint about the poor state of in-flight Internet service. After all, it's incredibly frustrating when you're on a deadline and unable get any work done because you can't even load the Ars CMS. But there is light at the end of the tunnel, as I discovered late last year. Gogo Air, which provides in-flight connectivity on most of the major US airlines, noticed one of my frustrated outbursts (on AA680, PHX to DCA, November 13 for the curious) and invited me to try out its latest service, a satellite-based system called 2Ku. Compared to the ATG4 system that most flyers are currently saddled with—including this author right now, currently on AA2617 at 37,000 feet—the difference is night and day.

Gogo Air provides in-flight Internet connectivity to most US passenger airlines (and quite a few international ones) and has been doing so since 2008. Originally that was with a cellular service called ATG—for Air-To-Ground—which leveraged the old Airfone cellular network. More recently Gogo Air upgraded that system to ATG4, bumping per-plane bandwidth from 3.1Mbps to 9.8Mbps. (For a much more indepth look at the state of in-flight Wi-Fi back in the day, check out this comprehensive feature from 2011.)

Read 11 remaining paragraphs | Comments

Hacker NewsStrikingly (YC W13) is hiring designers and devs in Shanghai
Comments
Hacker NewsBiotech booms in China
Comments
Hacker NewsGerman Union Steps Up Fight for ‘Modern’ 28-Hour Workweek
Comments
Hacker NewsApply to Y Combinator (Summer 2018 Batch)
Comments
Hacker NewsThe iOS Economy, Updated (2018)
Comments
Hacker NewsThe Navy built a warship for $440M. It's been stuck in ice since Christmas Eve
Comments
Hacker NewsRoot Cause of Reboot Issue Identified; Updated Guidance
Comments
Ars TechnicaAmazon Go debuts, and its prying cameras foil our shoplifting attempts

Enlarge / Amazon Go: The bottle. (credit: Sam Machkovech)

SEATTLE—A little more than one year ago, I tried, and failed, to sneak into Amazon Go. The pilot version of Amazon's first grocery store experiment advertised a first in the world of brick-and-mortar shopping: if you want to buy something, just pick it up, toss it in your bag, and walk out. A camera system watches you and uniquely tags every item you pick up, then the story automatically charges a pre-registered credit card for the purchases. No clerks, no check-out aisles.

Amazon's late-2016 announcement of this store was more about building buzz than letting the public in, however. Initially, it was limited only to Amazon employees. Worse, promises that the shop would open for average consumers in "early 2017" didn't come close to fruition, with insiders indicating to Ars that the store's camera-tracking system didn't hold up to larger testing scrutiny as anticipated. But with only 24 hours' notice, that changed on Monday. That same Seattle pilot shop—the one Amazon staffers refused to let us into in December 2016—finally opened its doors to anybody with a smartphone and the Amazon Go app.

Meaning, customers didn't even need an Amazon Prime membership. If you want to stroll into the world's first Amazon Go store, all you need is an Amazon account with valid credit card information and a working smartphone. Turns out, I had both of those, so I walked, bleary-eyed, into the shop shortly after it opened at 7am Pacific time on Monday.

Read 19 remaining paragraphs | Comments

Hacker NewsOil Demand Peaking by 2030 as Electric Vehicles Boom
Comments
Hacker NewsDr. Seuss’ Satirical Lesson on Nuclear Armament and the Absurdity of War
Comments
Hacker NewsHow the brain can lose and reclaim an abandoned mother tongue
Comments
Hacker NewsWhy Pain Is Necessary
Comments
Ars TechnicaTesla owner attempts Autopilot defense during DUI stop

Enlarge (credit: California Highway Patrol)

A San Francisco Tesla owner has learned the hard way that Tesla's Autopilot feature does not excuse getting behind the wheel while intoxicated. On Saturday, January 13, police discovered a man in his Tesla vehicle on the San Francisco–Oakland Bay Bridge. The San Francisco Chronicle reports that "the man had apparently passed out in the stopped car while stuck in the flow of busy bridge traffic at 5:30pm, according to the California Highway Patrol."

When police woke the man up, he assured officers that everything was fine because the car was "on autopilot." No one was injured in the incident, and the California Highway Patrol made a snarky tweet about it:

Needless to say, other Tesla owners—and people who own competing systems like Cadillac's Super Cruise—should not follow this guy's example. No cars on the market right now have fully driverless technology available. Autopilot, Supercruise, and other products are driver assistance products—they're designed to operate with an attentive human driver as a backup. Driving drunk using one of these systems is just as illegal as driving drunk in a conventional car.

Read 3 remaining paragraphs | Comments

Hacker NewsShow HN: HTML5 webcam pics to email with motion detection
Comments
Hacker NewsTurning Down a Blockchain Job Offer
Comments
Hacker NewsMicrosoft’s PowerShell Core Offers Cross-Platform Automation
Comments
Hacker NewsWhat happens when you grep the file you've redirected grep to?
Comments
Hacker NewsMeet the People Keeping Mold-A-Rama Alive
Comments
Ars TechnicaAmerica finally gets Honda’s best: The 2018 Civic Type R

Type R-ism has long been vacant in the American car world. Used but once before, on the holy Acura Integra Type R (sold as a Honda in Europe and Asia), Honda has reserved its sharpest tools for other markets. But that stops now: the new Civic Type R is the very first Honda to wear the R badge in America and it's here to fight its bigger and more established competitors. There's no mistaking how and where on the grand automotive scale to place the new Civic Type R, even for the casual layperson. With a prominent splitter, wing, roof spoiler, air-channeling lumps and vents, and the curious three exhaust tips, it's replete with boy-racer visuals.

But this boy arrives with the right tools. Tucked under the cornucopia of aero and design appendages sit some drop-dead serious bits of hardware that check all the boxes. But there's a far more elusive and important box that this car manages to check—one of mechanical harmony.

Read 11 remaining paragraphs | Comments

Sous le Ginkgo#670 - les poux

Et c'est parti pour deux semaines de combat acharné...
Hacker NewsFaster R-CNN: Down the rabbit hole of modern object detection
Comments
Hacker News219 days of postmarketOS
Comments
Hacker NewsHere’s How Scammers Are Using Fake News to Screw with Bitcoin Investors
Comments
Ars TechnicaToday’s the day you can go to Amazon Go—a store with no queues and no cash

Enlarge / "Let's go shopping!" "No, let's Amazon Go shopping." "Dave, I hate your puns." (credit: Sam Machkovech)

After over a year of testing, Amazon will open its convenience store with no checkout queues and no cashiers, dubbed Amazon Go, to the public. The allure of Amazon Go is customers' ability to enter the store, put their items into their own shopping bags, and walk out. Amazon Go has no human cashiers to interact with, as all transactions are made wirelessly through the Amazon Go app and your Amazon account.

The online retailer announced Amazon Go in December 2016, opening its single location to Amazon employees only. Customers have to scan the Amazon Go app on a turnstile-like entryway before they can go into the store and shop. Once inside, customers can pick up any items they want—food, drinks, essential items, and even alcohol—and leave when they wish, without standing in a checkout line or interacting with a cashier. One of the few store employees will have to check your ID if you purchase alcohol, but otherwise you can leave when you've finished shopping and your Amazon account will be billed for the items you chose.

Amazon Go works thanks to an outfitting of special cameras, shelf sensors, and the company's computer vision system that together monitor your actions in the store as well as the movement of items on shelves. The technology isn't fool-proof, though, and it can get confused when the store is crowded or when items get misplaced in the store. According to a Recode report, these instances pushed back the debut of Amazon Go to the public. The company originally planned to open the Seattle location of Amazon Go to the public early last year.

Read 1 remaining paragraphs | Comments

Hacker NewsResearchers implement 3-qubit Grover search on a quantum computer
Comments
Hacker NewsPyTorch, a year in
Comments
Hacker NewsJust 8 men own same wealth as half the world
Comments
Ars TechnicaRocket Lab makes it into orbit, nears commercial operations

Rocket Lab

Last Sunday afternoon in New Zealand, Rocket Lab successfully made it into orbit on just the company's second flight attempt. Launching from a new spaceport that juts into the South Pacific Ocean, the Electron rocket climbed into orbit and deployed its customer payloads at 8 minutes and 31 seconds after lifting off.

“Today marks the beginning of a new era in commercial access to space. We’re thrilled to reach this milestone so quickly after our first test launch,” Rocket Lab CEO and founder Peter Beck said in a statement. “Our incredibly dedicated and talented team have worked tirelessly to develop, build and launch Electron. I’m immensely proud of what they have achieved today.”

Read 4 remaining paragraphs | Comments

Hacker NewsBitcoin.org removes “fast” and “low fees” from Bitcoin description page
Comments
Hacker NewsBeyondCorp: How Google Ditched VPNs for Remote Employee Access
Comments
Hacker NewsTwist, startup behind AirPlay-powered smart home light, abruptly shuts down
Comments
Hacker NewsHow a single line of computer code put thousands of innocent Turks in jail
Comments
Hacker NewsShow HN: Revenue Forecasting for SaaS Apps
Comments
Planet PostgreSQLFederico Campoli: Migrating MySQL to PostgreSQL with minimal downtime
When I started writing pg_chameleon I had in mind a tool making simple to migrate from different dbms engines to PostgreSQL.

The name was inspired by the Culture Club's Karma chameleon song which suggested me the idea of a tool capable to move data between different database technologies.

Then tool's scope changed becoming a replica system rather a migration system but I've kept the name as I think the idea is still valid.

However, during the development of the now discontinued version 1 I needed to migrate a MySQL database rather replicating it. Therefore I added a command to detach the replica, allowing the migration to happen with very minimal downtime.


Prerequisites


pg_chameleon requires python to be at least 3.3; in this example we'll use python 3.6.

The the tables during the initial copy are locked one by one.
So, probably, the best way to setup the MySQL-PostgreSQL replica is to use a cascading MySQL as a source in order to minimise the downtime.

This example use the test database sakila as MySQL source.
The instructions on how to install it are available here.

Setup the virtual environment

Let's start configuring a virtual environment.
In a shell run the following commands.

thedoctor@deadpool:~$ python3 -m venv migration
thedoctor@deadpool:~$ source migration/bin/activate





Then let's upgrade pip as a good measure

(migration) thedoctor@deadpool:~$ pip install pip --upgrade
Requirement already up-to-date: pip in ./migration/lib/python3.6/site-packages


Now let's install pg_chameleon.

(migration) thedoctor@deadpool:~$ pip install pg_chameleon
Collecting pg_chameleon
  Downloading pg_chameleon-2.0.2.tar.gz (58kB)
    100% |████████████████████████████████| 61kB 1.2MB/s
...

...
...
  Running setup.py install for mysql-replication ... done
  Running setup.py install for PyYAML ... done
  Running setup.py install for tabulate ... done
  Running setup.py install for rollbar ... done
  Running setup.py install for pg-chameleon ... done
Successfully installed PyMySQL-0.8.0 PyYAML-3.12 argparse-1.4.0 certifi-2018.1.18 chardet-3.0.4 daemonize-2.4.7 idna-2.6 mysql-replication-0.15 pg-chameleon-2.0.2 psycopg2-2.7.3.2 requests-2.18.4 rollbar-0.13.17 six-1.11.0 tabulate-0.8.2 urllib3-1.22
(migration) thedoctor@deadpool:~$



As last step let's create the configuration directory running the set_configuration_files command.

(migration) thedoctor@deadpool:~$ chameleon set_configuration_files
creating directory /home/thedoctor/.pg_chameleon
creating directory /home/thedoctor/.pg_chameleon/configuration/
creating directory /home/thedoctor/.pg_chameleon/logs/
creating directory /home/thedoctor/.pg_chameleon/pid/
copying configuration  example in /home/thedoctor/.pg_chameleon/configuration//config-example.yml


MySQL setup

Now let's configure the databases for the migration.
The MySQL server's configuration depends on if we are using the master or a cascading replica.

If we are using the master, then the my.cnf file should contain the following rows under the [mysqld] entry.

binlog_format= ROW
log-bin = mysql-bin
server-id = 1
binlog-row-image=FULL

In the case of a cascading replica the only difference is the server-id which should be different from the value 1 and unique within the replication cluster.
A reasonable value could be a number composed by the date and a counter, similarly to the DNS serial numbering (e.g. 201801221).

We also need to set the parameter log-slave-updates = 'on'.
This parameter will tell the slave to log the changes into its local binary log location. The cascading replica configuration will be something like this.

binlog_format= ROW
log-bin = mysql-bin
server-id =201801221
binlog-row-image=FULL
log-slave-updates = On

Appliying these settings requires the database restart .


Database users setup

Now we are ready to setup the users on MySQL and postgres.
Let's give them the same name usr_replica. 

On MySQL the user needs the full access to the replica database  plus the RELOAD, REPLICATION CLIENT and REPLICATION SLAVE on the cluster.

CREATE USER usr_replica ;
SET PASSWORD FOR usr_replica=PASSWORD('replica');
GRANT ALL ON sakila.* TO 'usr_replica';
GRANT RELOAD ON *.* to 'usr_replica';
GRANT REPLICATION CLIENT ON *.* to 'usr_replica';
GRANT REPLICATION SLAVE ON *.* to 'usr_replica';
FLUSH PRIVILEGES;


On PostgreSQL we shall create a normal user and a database which owner is the replication user.

CREATE USER usr_replica WITH PASSWORD 'replica';
CREATE DATABASE db_replica WITH OWNER usr_replica;



 Configuring pg_chameleon


After the configuration directory is created we need to step into ~/.pg_chameleon/confiuration and copy the config-example.yml in a new configuration file.

(migration) thedoctor@deadpool:~$ cd .pg_chameleon/configuration/
(migration) thedoctor@deadpool:~/.pg_chameleon/configuration$ ls
config-example.yml
(migration) thedoctor@deadpool:~/.pg_chameleon/configuration$ cp config-example.yml migration.yml
(migration) thedoctor@deadpool:~/.pg_chameleon/configuration$


The configuration file's details are described here.
In a minimal configuration the parts we need to setup are the PostgreSQL database connection and the mysql source setup.


pg_conn:
    host: "destination_host"
    port: "5432"
    user: "usr_replica"
    password: "replica"
    database: "db_replica"
    charset: "utf8"



The config-example.yml comes with the mysql and pgsql sources.
As we are configuring only the mysql part we'll remove the pgsql entry.

sources:
    mysql:
        db_conn:
            host: "source_host"
            port: "3306"
            user: "usr_replica"
            password: "replica"
            charset: 'utf8'
            connect_timeout: 10
        schema_mappings:
                sakila: sch_sakila
        limit_tables:
        skip_tables:
        grant_select_to:
            - usr_migration
        lock_timeout: "120s"
        my_server_id: 201801222
        replica_batch_size: 10000
        replay_max_rows: 10000
        batch_retention: '1 day'
        copy_max_memory: "300M"
        copy_mode: 'file' 
        out_dir: /tmp
        sleep_loop: 1
        on_error_replay: continue
        type: mysql

The part to pay attention is the schema_mappings key.
In this area is possible to map multiple schemas from mysql to postgresql.
Once the source is registered the schema mapping is stored in the replica catalogue and no longer read from the configuration file.

Any change of schema_mappings made into the configuration file shall be applied to the replica catalogue with the command update_schema_mappings.

Another section that needs configuration is the type_override.
The given example overrides the tinyint(1) into boolean for all the tables.

type_override:
    "tinyint(1)":
        override_to: boolean
        override_tables:
            - "*"


As this is quite the norm on the MySQL databases it could be the need to limit this override to specific tables.
With a boolean override like this not valid values will be skipped during the initial copy and any row with invalid data will cause the replica for the affected table to stop.

If there's no need for type override is possible to disable leaving the type_override: key without entries.

The entry grant_select_to lists the database users or roles which have usage and select privileges over  the replicated schema.

Configuring the replica

Now we are set for the replica so, after activating the virtualenv we shall install the replica catalogue and add the mysql source.

chameleon create_replica_schema
chameleon add_source --config migration --source mysql

If the log_dest is set to file you'll see the output logged in the directory log_dir.

You can check the source status with the command

chameleon show_status --config migration

  Source id  Source name    Type    Status    Consistent    Read lag    Last read    Replay lag    Last replay
-----------  -------------  ------  --------  ------------  ----------  -----------  ------------  -------------
          1  mysql          mysql   ready     Yes           N/A                      N/A

== Schema mappings ==
Origin schema    Destination schema
---------------  --------------------
sakila           sch_sakila

== Replica status ==
---------------------  -
Tables not replicated  0
Tables replicated      0
All tables             0
Replayed rows
Replayed DDL
Skipped rows
---------------------  -


Now we are ready to initialise the replica with the command init_replica.

chameleon init_replica --config migration --source mysql
Init replica process for source mysql started.

When displaying the source status the column name will change to initialising and then to initialised. If the rollbar_key and rollbar_env are set then a message to rollbar will be sent when the process is complete.

To start the replica just run

chameleon start_replica --config migration --source mysql
Starting the replica process for source mysql
chameleon show_status --config default --source mysql
  Source id  Source name    Type    Status    Consistent    Read lag    Last read    Replay lag    Last replay
-----------  -------------  ------  --------  ------------  ----------  -----------  ------------  -------------
          1  mysql          mysql   running   No            N/A                      N/A

== Schema mappings ==
Origin schema    Destination schema
---------------  --------------------
sakila           sch_sakila

== Replica status ==
---------------------  --
Tables not replicated   0
Tables replicated      20
All tables             20
Replayed rows
Replayed DDL
Skipped rows
---------------------  --


Migrating the database

Before migrating the database to PostgreSQL we should wait for the replica to reach consistent status and there are no tables with the replica disabled listed in the show_status output.

After stopping the writes on mysql we should wait that the replay lag drops to zero. This will ensures all the changes have been replayed on PostgreSQL.
Without writes on the MySQL source the read lag will increase as no events will come trough the replica.

We are then ready to detach the replica.
We are using the debug option to get a verbose output on the standard output.
chameleon detach_replica --config default --source mysql --debug
Detaching the replica will remove any reference for the source mysql.
 Are you sure? YES/No

YES
2018-01-22 12:28:25 MainProcess DEBUG pg_lib.py (608): Changing the autocommit flag to True
2018-01-22 12:28:25 MainProcess DEBUG pg_lib.py (2598): Collecting schema mappings for source mysql
2018-01-22 12:28:25 MainProcess INFO mysql_lib.py (297): retrieving foreign keys metadata for schemas 'sakila'
2018-01-22 12:28:25 MainProcess DEBUG pg_lib.py (2598): Collecting schema mappings for source mysql
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.actor_actor_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.address_address_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.category_category_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.city_city_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.country_country_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.customer_customer_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.film_film_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.inventory_inventory_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.language_language_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.payment_payment_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.rental_rental_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.staff_staff_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.store_store_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.test_partition_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.test_partition2_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.test_tin4y_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (711): resetting the sequence  sch_sakila.test_tiny_id_seq
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_address_city_0 on table sch_sakila.address
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_city_country_1 on table sch_sakila.city
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_customer_address_2 on table sch_sakila.customer
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_customer_store_3 on table sch_sakila.customer
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_film_language_4 on table sch_sakila.film
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_film_language_ori_5 on table sch_sakila.film
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_film_actor_actor_6 on table sch_sakila.film_actor
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_film_actor_film_7 on table sch_sakila.film_actor
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_film_category_cat_8 on table sch_sakila.film_category
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_film_category_fil_9 on table sch_sakila.film_category
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_inventory_film_10 on table sch_sakila.inventory
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_inventory_store_11 on table sch_sakila.inventory
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_payment_customer_12 on table sch_sakila.payment
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_payment_rental_13 on table sch_sakila.payment
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_payment_staff_14 on table sch_sakila.payment
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_rental_customer_15 on table sch_sakila.rental
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_rental_inventory_16 on table sch_sakila.rental
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_rental_staff_17 on table sch_sakila.rental
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_staff_address_18 on table sch_sakila.staff
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_staff_store_19 on table sch_sakila.staff
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_store_address_20 on table sch_sakila.store
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (740): creating invalid foreign key fk_store_staff_21 on table sch_sakila.store
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_address_city_0 on table sch_sakila.address
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_city_country_1 on table sch_sakila.city
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_customer_address_2 on table sch_sakila.customer
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_customer_store_3 on table sch_sakila.customer
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_film_language_4 on table sch_sakila.film
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_film_language_ori_5 on table sch_sakila.film
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_film_actor_actor_6 on table sch_sakila.film_actor
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_film_actor_film_7 on table sch_sakila.film_actor
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_film_category_cat_8 on table sch_sakila.film_category
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_film_category_fil_9 on table sch_sakila.film_category
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_inventory_film_10 on table sch_sakila.inventory
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_inventory_store_11 on table sch_sakila.inventory
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_payment_customer_12 on table sch_sakila.payment
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_payment_rental_13 on table sch_sakila.payment
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_payment_staff_14 on table sch_sakila.payment
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_rental_customer_15 on table sch_sakila.rental
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_rental_inventory_16 on table sch_sakila.rental
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_rental_staff_17 on table sch_sakila.rental
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_staff_address_18 on table sch_sakila.staff
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_staff_store_19 on table sch_sakila.staff
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_store_address_20 on table sch_sakila.store
2018-01-22 12:28:25 MainProcess INFO pg_lib.py (750): validating fk_store_staff_21 on table sch_sakila.store
2018-01-22 12:28:25 MainProcess DEBUG pg_lib.py (1787): Deleting the source mysql
2018-01-22 12:28:25 MainProcess DEBUG pg_lib.py (628): There is already a database connection active.


The procedure resets all the sequences using the max values of the depending fields.
All the foreign keys are generated from the MySQL information schema and created as invalid. A second run validates them. If any error occurs the key is left invalid but at least is enforced.

Finally the source is dropped from the replica catalogue.

Wrap up

With this method is possible to keep the writes on MySQL until the last moment and then switch to PostgreSQL after the detach is finished.

The sequence reset and the foreign key validation are the biggest pain in this process but is very likely the time required is lesser than the time used for dumping the database from MySQL and reloading it into PostgreSQL.

Currently there is no functionality for setting automatically the write permissions on the detached schema.


The foreign keys are created without any clause as I haven't figured out how to get that from the information_schema.

Probably I will add both improvements it in a future minor release.

That's all.
Thank you for reading.
Hacker NewsI trained an AI to copy my voice and it scared me silly
Comments
Hacker NewsObjects larger than the wavelength of sound in an acoustic tractor beam
Comments
Hacker NewsWhy are doors on British trains so difficult to open?
Comments