American Republican legislators have begun aiming their sights on a major policy initiative: the nation's tax code. Any changes will certainly impact the American technology sector, but before getting to that possible impact, there's the matter of the GOP's publicity campaign on the matter.
On Wednesday afternoon, the GOP showed that it could use some help in its attempts to make its sales pitch look "hip."
In a blog post titled "What Do The Legend of Zelda and the American Tax Code Have In Common," House Republicans originally wrote:
The documents included internal papers published by journalists at InsideClimate News as well as 50 “peer-reviewed articles on climate research and related policy analysis” written by ExxonMobil researchers. The oil and gas company made the internal papers public and challenged anyone to “read all of these documents and make up your own mind,” accusing journalists of cherry-picking data.
Geoffrey Supran and Naomi Oreskes, from Harvard's Department of the History of Science, took up that challenge, comparing the information in the documents cited by ExxonMobil against the information conveyed in the publicly-available advertorial columns published by the company on anthropogenic (or human-caused) climate change in the New York Times. They found that “83 percent of peer-reviewed papers and 80 percent of internal documents acknowledge that climate change is real and human-caused, yet only 12 percent of advertorials do so, with 81 percent instead expressing doubt.”
If the headline on this article made your eyes burn with fire, and your fingers twitch to comment without reading further, then you're in the majority. The relationship between Silicon Valley's tech industry and economic inequality in the Bay Area is an incendiary issue, as civic tech leader Catherine Bracy is all too aware. She came to Ars Technica Live to talk about her vision for a future where people in Oakland celebrate when a new tech company comes to town. With her startup TechEquity Collaborative, she's showing techies what they can do to help their neighbors benefit from the tech economy as much as they have.
Bracy's first message to the packed crowd was that we shouldn't blame techies for the housing crisis. She said she'd had to overcome her own prejudices to make that realization. When Uber announced they were buying the historic Sears Building in Oakland, where Bracy lives, she said she had to struggle not to get angry. She worried that her neighborhood would be less brown, and that there would be a wave of housing displacement like the one San Francisco has already experienced. But after years of working on civic-minded tech projects like Code for America, and founding nonprofit TechEquity several years ago, she's come to a new understanding.
The Federal Trade Commission has formally allowed Amazon’s acquisition of Whole Foods to go forward.
According to a statement released Wednesday by Bruce Hoffman, the acting FTC director: "Based on our investigation, we have decided not to pursue this matter further. Of course, the FTC always has the ability to investigate anticompetitive conduct should such action be warranted."
Last month, there had been some public opposition to the deal. Back in June, the online retail giant announced it would acquire Whole Foods Market for approximately $13.7 billion.
Given that, I am surprised that it is difficult to find a comprehensive list of, "these sites are fucking nazis and you should blacklist them from your ads." Here's one. It seems like this list should be a lot longer. Let me know if you have a better resource.
When one searches for said blacklist, what one finds instead are are ten thousand breathless articles where some brand's PR flunky is quoted saying "we swear we are trying not to advertise with nazis, but the ads keep showing up there anyway and we can't make it stop." So that's awesome.
A New York City father and son have been arrested and indicted on allegations of selling fentanyl and oxycodone on the underground drug website AlphaBay, which was seized and closed by federal law enforcement in July.
The indictment comes days after six Californians were indicted on similar charges of drug trafficking on AlphaBay, suggesting that federal law enforcement is now ready to start prosecuting at least some of the dealers that used the notorious site.
According to a newly issued criminal complaint, Michael Luciano admitted in July to selling the opioid drugs, with the help of his son Philip Luciano, out of his Staten Island home.
That's the overwhelming feeling I got after a test drive with the Galaxy Note8. Samsung's earlier flagship for 2017, the Galaxy S8—specifically the Galaxy S8+—is so close to the Note8 I'm not sure why anyone would wait the five months of lag time between the two devices.
In a sign of the soaring demand for zeroday attacks that target software that's becoming increasingly secure, a market-leading broker is offering serious cash for weaponized exploits that work against Signal, WhatsApp, and other mobile apps that offer confidential messaging or privacy.
Zerodium, the Washington, DC-based broker that launched in 2015, said on Wednesday that it would pay $500,000 for fully functional attacks that work against Signal, WhatsApp, iMessage, Viber, WeChat, and Telegram. The broker said it would start paying the same rate for exploits against default mobile e-mail apps. Those are among the highest prices Zerodium offers. Only remote jailbreaks for Apple's iOS devices fetch a higher fee, with $1.5 million offered for those that require no user interaction and $1 million for those that do. The jailbreak fees were announced in September 2016 and September 2015, respectively.
"Overall prices are trending up—and quite significantly in many cases, and there's an increased focus on mobile," Adam Caudill, a senior application security consultant at AppSec Consulting, told Ars. "The new $500k targets for messaging and default e-mail apps show what a priority attacking individuals via their devices has become (which makes sense, given the recent increase in state-sponsored malware targeting mobile devices via SMS and the like)."
If you’ve ever wondered whether psychotherapy achieves meaningful, long-term change in a person’s life, wonder no more: combined evidence from multiple studies suggest that it does. A meta-analysis published recently in Psychological Bulletin reports that a variety of different therapeutic techniques results in positive changes to personality, especially when it comes to neuroticism, that last over a considerable period of time.
Personality is, as your intuition might tell you, relatively stable—people who start out gregarious and adventurous tend to stay gregarious and adventurous throughout their lives. Assessments of people’s personality traits taken at different times tend to agree pretty well with each other. But that doesn’t mean personality is static: personal growth, life experiences, and age all play their part, and people’s personalities do change somewhat throughout their lives—usually for the better.
An OCEAN of change
But it can be tricky to work out precisely what is being evaluated in measures of personality like the “Big Five” of Openness to experience, Conscientiousness, Extraversion, Agreeableness, and Neuroticism ("OCEAN"). Any personality questionnaire will come up with metrics that capture both someone's stable, long-term tendencies (their traits), as well as how they are feeling in a given moment or phase in their life (their state). So, it’s not enough to find that therapy brings about personality changes—it’s also necessary to figure out how deep those changes go.
Anybody who knows that Donald Trump is president of the United States probably knows that his Twitter feed is his mouthpiece to the world. He criticizes global leaders, threatens war, berates the media, and, well, you name it. It's no understatement to suggest that the nation's 45th chief executive has assumed the Twitter Presidency.
Now there is a movement afoot to silence the @realdDonaldTrump handle, which has 36.6 million followers and has tweeted more than 35,000 times. Despite allegations that Trump is breaching Twitter's terms of service for using the service to threaten violence, Twitter isn't about to kick off its most high-profile tweeter.
So a former CIA undercover operative is trying to do what Twitter won't: end the Twitter Presidency. Although her plan most likely will never come to fruition, Valerie Plame Wilson has started a GoFundMe page to raise $1 billion, which would make her Twitter's largest shareholder and give her great monetary influence over Twitter's policies.
The Windows 10 Fall Creators Update, aka version 1709, is due to be finished next month. Accordingly, the builds released to the Windows Insider Program are now heading toward stabilization, with bug-fixing the priority.
This has a somewhat greater significance than was the case for prior Windows 10 updates because of the "Skip Ahead" scheme Microsoft introduced a few weeks ago. Most members of the Insider Program will continue to receive Redstone 3 builds and head toward stabilization and finalization for the Fall Creators Update release. Those who opted in to the (limited) Skip Ahead group, however, will imminently be switched to a different Windows branch. This branch will begin development of the Redstone 4 release that's due in March 2018.
Once the Fall Creators Update release is out, all Insiders will be moved to the Redstone 4 branch, and the program will continue as before.
Major home Internet providers in the US don't typically expand into each other's territory, but this week, AT&T said it is launching high-speed Internet in parts of New York City and other major metro areas outside of its traditional wireline footprint.
The new service is for apartment and condominium buildings, so don't expect to get it if you live in a single-family house. It's also only available in cases where AT&T has gotten access into buildings, which is often a problem for competing ISPs because of exclusive arrangements between providers and landlords. But for some consumers, the new AT&T launch could provide some much-needed competition.
AT&T's new deployment uses G.fast, a technology that relies on fiber deployments into neighborhoods and copper wires to make the connection inside each building. But instead of old phone lines, AT&T said it is using coaxial cables to make the final connection to consumers.
A patent-licensing entity that sued the five largest cell phone carriers has seen its biggest victory slip away.
Prism won a $30 million verdict against Sprint in 2015, when a jury found that Sprint violated US Patents No. 8,127,345 and 8,387,155, both of which describe methods of "managing access to protected computer resources." According to the complaint (PDF), filed in 2012, Sprint's Simply Everything Plan and Everything Data Plan were both methods of "controlling access to Sprint’s protected network resources" and thus infringed the patents.
Earlier this month, though, US District Judge Lyle Strom ruled (PDF) that Prism won't be able to collect on its verdict. Prism used the same patents to take on T-Mobile, which defeated the patents and prevented Prism from collecting the Sprint verdict as well.
If All in Hyrule Are Like You, Maybe You’ll Do All Right: Twilight Princess and Overcoming Individualism
[This essay was funded through Patreon & Medium under the ZEAL project. ZEAL aims to provide high quality criticism of rarely discussed games and comics, and showcase the talents of exciting new writers and artists. For details and information on how to donate, please check out our Patreon, or consider becoming a Premium member for exclusive content!]
It’s seven in the morning sometime in June 2007, and I’m the only one awake in what feels like the whole world. I’m up before my parents are, having slipped past their room on my way downstairs to settle into place on uncomfortably itchy carpet in front of the TV. I’m double-checking that the sound is turned all the way down so they won’t wake up to a long, drawn-out wolf howl that comes at the end of the opening to Legend of Zelda: Twilight Princess. I’m eleven years old, and I’m probably about thirty hours into the game without finishing it yet. This is totally fine with me. I never want it to end.
No, I wasn’t bad at Zelda games, despite how the many I owned had 50+ hours clocked in before they were finished. Rather, I grew up taking things slowly, from books to movies to walks down my suburban street, and my experience of Twilight Princess was no different. I enjoyed the overworld so much I would often spend a couple of hours a day just swimming in Lake Hylia, riding across Hyrule Field, or perhaps spending some time at the fishing hole if I felt like getting some action in. When I felt like I needed some company, I visited Kakariko Village or Castle Town, visiting the yetis in their mountain mansion, or ending up at Telma’s tavern, enjoying the presence of the people there. I didn’t need to talk to them — I knew what they would say, their dialogue fixed and static, their personalities only revealed in full at plot-relevant points. That didn’t stop me from enjoying their company, though, though it took a long time to earn their trust.
Though their group is never named in-game, this group of characters, comprising of
· Rusl, Link’s father figure from Ordon Village
· Ashei, the daughter of a Hyrulian knight
· Shad, a dedicated researcher of Hyrule’s history
· Telma, bartender, bar owner, and a surprisingly skilled wagon driver;
· Louise, Telma’s prissy but kind white cat;
· and Auru, a veteran knight who is well-versed in weaponry and the royal family’s history
make up the Resistance. Like Link, these five Hylians and one cat are working tirelessly to figure out a way to stop the spread of the Twilight. Rather than give in to the tyrannical monarchy under Zant, this group functions much like any resistance group does in such a story: by plotting the usurper’s demise. Prior to the advent of Link’s venture into the game’s plotline, its clear they were missing a component, someone who could add another set of skills to their Swiss Army knife of an activist team. This is, of course, where Link comes in, the eponymous hero of his own story — and, now, theirs.
However, one person alone does not build community. Accepting a new member, trusting a new face, takes time, and Twilight Princess pits Link against more grueling efforts than ever, even though at this point, the player is into the second half of the game. Despite Telma’s preexisting trust in Link after she witnesses him rescue Ilia and the Ordon Village children in Kakariko, the others are unwilling to accept his presence. In times of hardship, the thought of accepting outsiders is often the last thing on anyone’s mind. Link’s presence is suspicious at worst and a joke at best — ironically, Ashei mocks him for his outfit. She thinks he’s playing dress-up, attempting to imitate the guise of the legendary hero. Link, as a character, is not unused to this. As a new player of Zelda, however, perhaps these words would sting. The last thing anyone — including Link — wants is rejection. But over and over again, that is what he — and you, the player — must face.
Twilight Princess shows, at its most ironic point, the Resistance realizing they should apologize to Link as he hides in the rafters above them in wolf form. To fall then is to be kicked back out of the bar by a Goron bouncer in a fade-out that implies a violent struggle between Link and those below. Link, of course, doesn’t express any upset at this — he can’t. At this point, completing the task at hand is more important than trying to convince others he isn’t who he appears to be. It’s nice to think that they would be willing to accept him if they knew about his wolfish alter-ego, but there’s something a little unrealistic about that in a game that is pretty good at keeping to a realistic narrative. The Resistance have reactions that one would expect in a tense a situation as the fate of their known world: anger, determination, disdain, suspicion, and, below it all, curiosity.
Though rich, the story arc with the members of the Resistance is unfortunately brief and choppy. It’s interspersed with lengthy dungeons and various side-quests, meaning it’s easy to focus on the tasks ahead rather than the burgeoning sense of belonging that’s in development for Link. Who, then, is ideal in showing this development of acceptance and reliance on community than Midna? As Link’s constant companion, she goes through as much as he does, but her life spent literally in the his shadow means she doesn’t have access to the same level of interaction that Link does — she has to watch it all unfold. All she has is Link. At first, among her insults and demands, she seems to try to pressure him into going it alone. Midna’s connection (or lack thereof) to community works as a foil to Link’s — she doesn’t understand why he would seek out the help of people who don’t seem to appreciate his presence. In fact, she seems totally abject to being around others at all. Her exile by Zant and the lack of support she has in the world of light leaves her bitter, manipulative, and lonely.
She doesn’t desire Link for his company, but rather for his use. She minces her words, mocks Link and the other Hylians important to him, and seems to care little for the struggle of those life forms that cannot survive in the Twilight. It’s easy to interpret this as part of her trickster-like personality, but the reality of the situation is, she is herself hurt and lonely, torn from all she knew and held dear. While it can be presumed that Link’s rupture strengthens his resolve, Midna’s seems to have left her feeling weak and powerless, though she certainly doesn’t show it. Her wit is as sharp as Link’s bite, making them a good match despite their different views on working with others — at first.
Leaving loneliness behind is an important part of finding community, too. Often, individualism is toted as the be-all, end-all state of success: if you can do it on your own, you’re better than those who cannot. But this is inherently untrue. If I had known, as a child, I would have been much better off with a core group of friends, I would have sought that out to no end. Exiting loneliness seems to be a lifelong process when you grow up without a strong sense of community. Midna’s development as a character in particular reflects this. Her determination to use Link rather than work with him is soon discarded after she witnesses Link perform selfless act after selfless act in order to help those around him. She doesn’t understand why he would do anything if it wasn’t solely for his own benefit, but she soon comes to terms with his desire to be selfless. It isn’t Link who shows her the ultimate sacrifice, though. In a touching turn of events, the crux of this development occurs when Zelda seemingly sacrifices herself so that Midna can live.
From then on, Midna felt she no longer was doing this seemingly impossible task alone — she had found at least two others willing to fight for her life, among countless other tasks. Although it may have its own TVtropes page, the concept of ‘family of choice’ or ‘found family’ is a very real and important aspect in the lives of many marginalized people, including, of course, queer and trans people like myself. Rejected by parents or guardians, ostracized by siblings, abandoned by former friends — these are just a few scenarios that LGBTQ+ people, especially teenagers and young adults, often face. Twilight Princess may not ‘queer’ its narrative in this way, but as a queer and trans person, I can’t help but relate my own experiences as they occurred in the ten years since I played the game for the first time.
What offered me was a connection to a community when there was none available to me. A ragtag resistance organization of social outcasts depending on the questionable abilities of a teenage boy who can turn into a wolf (who himself is relying on the aid of a exiled queen in the form of an impish, sharp-witted shadow) was my ideal for a community, a friend group, a family. I don’t think I will ever be able to forget what Twilight Princess offered me: an understanding that strength and success are easier when there are others working with you. To say that these characters literally were my community would not be wrong. At the time, I had little else. I related deeply to Link and Midna’s desperate attempt to survive and save the worlds they inhabit because I felt connected to those worlds as well. It’s just good storytelling, in the end –it creates a sense of thrilling urgency by capitalizing on the player’s attachment to the fictional people in a world forever teetering on destruction, living in stasis until the game is started up again and the adventure continues. Being unattached to the people and places, to the cities and towns, to the communities central to the success of the hero’s efforts would mean certain death for them all — or, at least, a disinterest in the game itself. Twilight Princess had its flaws, but the effort that went into fleshing out character roles to go beyond those of the average NPC still strikes me. Link may be the hero, but his success cannot be his alone.
Without the Resistance helping to storm the castle leading to the final battle with Ganon, it just wouldn’t be possible. Twilight Princess shows its player that much in a striking cutscene, wherein the group of them combine their strengths and talents to clear away a hoard enemies so that Link can progress through the castle’s labyrinthine interior with ease. Earning their trust and aid didn’t require leading them — rather, it entailed bonding with them. Like a folk tale, Twilight Princess wove together the stories of the Resistance by amping up not their special powers, superior intellect, or worldly importance: rather, the focus was on a collective effort and desire to help. Even so, at the end of the game, everyone, including Link, takes different paths. The future is uncertain, but the impact of the final scenes aren’t meant to come from the characters — it is felt in the player, who watches as everyone they relied on to seal away Hyrule’s evil once more part ways and return to their lives during the end credits. Worry, elation, relief, happiness — it’s all a part of saying goodbye to people important to you, knowing that yes, of course they’re still out there, somewhere, ready to return to aid you during difficult times. Even Midna’s final words reflect this as she returns indefinitely to the Twilight –
Our own Mark Walton plays Destiny 2 at 4K and 60 fps on PC.
Here at Ars we spenda lotof timetalkingabout how developers deal with the trade-offs between resolution, frame rate, graphical detail, and simulation complexity they face at the top end of modern console and PC hardware. Quite often, the first-blush "wow factor" of more pixels and higher frame rates wins out in this constant balancing act. For Destiny 2, though, Executive Producer Mark Noseworthy says the team prioritized the complexity of the game itself over hitting a frame rate higher than 30fps.
In a Twitter thread back in June, Noseworthy said that the CPU limits on current consoles mean the game had to scale back to 30fps "to deliver D2's AI counts, environment sizes, and # of players." In the latest issue of Edge magazine (excerpted by WCCFTech), Noseworthy expands on the reasoning behind that choice:
It’s about the simulation of the Destiny world. Thirty AI at once, large open spaces, six players, sometimes with vehicles, and dropships coming in; that’s where we’re using the CPU.
Could we make a Destiny game that ran at 60fps? Yes, but the space would be smaller, it would be less cooperative, and there’d be fewer monsters to shoot. That’s not the game we want to make.
First and foremost, we’re trying to make an incredible action game. We don’t feel we’ve been held back by the choices we’ve made about world simulation versus frame rate; in fact, we think we’re offering a player experience you can’t have elsewhere because of the choices we’re making.
Put like that, the trade-off doesn't sound like a bad one. Yes, a game that's locked to 30fps looks markedly worse than one running at 60fps or more, all things being equal. The resulting lack of smoothness is especially noticeable in a reflex-based shooting game like Destiny 2 (though the server's internal tick rate has arguably more impact on how the game feels). That said, a smoother Destiny 2 with fewer simultaneous enemies and fewer player characters in smaller battle locales would probably be noticeably worse to play, too. As long as the game can run steadily at a playable 30 frames per second, without dips, that sounds like a perfectly acceptable trade.
Linux uses an
(OOM), which is designed to kill processes:
If your memory is exhaustively used up by the processes to the extent which can possibly threaten the stability of the
system, then the OOM killer comes into picture. It is the task of the OOM killer to kill the processes until enough
memory is freed for the smooth functioning of the rest of the process.
While killing processes is never good, it is better than having the system halt due to memory exhaustion. Sometimes the OOM
kills Postgres, and that isn't a good thing either. This
thread explains how to use ulimit to cause Postgres
sessions that consume a lot of memory to fail due to excessive memory requests. This avoids having them continue and be killed by the
OOM killer, which causes the
server to restart. The Postgres
documentation also explains the behavior of
the OOM killer.
If you watch Top Gear, you'll know the Ariel Motor Company. It's the British maker of the Atom, a mid-engined assortment of scaffolding that was dreamt up as a modern answer to the Lotus/Caterham Seven—the same car that gave Jeremy Clarkson an epiglottis full of bees. Ariel also makes the Nomad, an off-road version of the Atom that featured in Matt Le Blanc's Top Gear debut.
Both of those vehicles are utterly bonkers, stripped down to the very essence of a car but overloaded with excitement. Which makes us rather excited about the fact that the next four-wheeled thrill ride to emerge from its Somerset factory is going to be an electric vehicle.
NEW YORK CITY—We're live from New York, where Samsung has taken the wraps off its new flagship device, the Galaxy Note8.
Samsung changed everything about the Galaxy S line earlier this year, and those changes are all making the jump to the bigger Note model. You get an extra-tall display with on-screen navigation buttons and slim bezels. The fingerprint reader has been moved to the back, next to the camera components. There's also an iris scanner, a dedicated hardware button for Samsung's "Bixby" voice assistant, and compatibility with Samsung's "Dex" desktop dock.
So what is actually different from the Galaxy S8, which launched almost five months ago? Well, first, it's slightly bigger. While the Galaxy S8+ topped out at 6.2-inches, the Note 8 bumps up to 6.3-inches. On the back there's Samsung's first dual-camera design, pairing a wide-angle camera with a telephoto one. Both have optical image stabilization. The Note8 has a more squared-off design, which leaves a bit more room internally for storage of the Note line's trademark S-Pen stylus, itself another addition over the S8.
The biggest thing in aerospace these days is the trend toward small things, from small satellites to small satellite launch vehicles like those under development by Rocket Lab, Virgin Galactic, and Vector Space Systems. Now a new microsatellite company, ICEYE, says it is moving forward with development and deployment of its synthetic-aperture radar technology.
On Wednesday morning, the Finland-based company will announce that it has raised $13 million in a new round of funding, including investments from space capital firms such as True Ventures, Lifeline Ventures, Space Angels, and Draper Associates. Since its founding in 2015, the company has raised $18.7 million.
In an interview with Ars, the company's chief executive and cofounder, Rafal Modrzewski, said ICEYE plans to launch its technology within the next 12 months. It intends to begin the launch of a full constellation by 2019. "For the first two years we were mainly a technology company, and we were working with customers to find their needs," he said. "Now we have matured the idea."
Early Wednesday morning, SpaceX founder Elon Musk posted a photo of the spacesuit that will be used by astronauts flying aboard the company's Dragon spacecraft, perhaps as early as next year. It is white, and looks futuristic.
In his Instagram post, Musk added that this suit was not a mock-up but rather a fully functional unit. "Already tested to double vacuum pressure," he wrote. "Was incredibly hard to balance aesthetics and function. Easy to do either separately." (Double vacuum pressure simply means the suit was probably inflated to twice the pressure of sea level and then put into a vacuum chamber).
Musk gave no other technical information about the suit. Most strikingly, it is white, in contrast to the very blue spacesuits unveiled by Boeing in January.
Uber is making additional changes to its driver-side app by allowing drivers to set more destinations and offering "long trip notifications" to tell a driver when a rider is requesting a ride that's 45 minutes or longer.
The company will also stop penalizing drivers who turn down trips. Previously, when a driver turned down potential trips, it could affect promotions and account standing.
The changes are part of a process the company is calling "180 Days of Change," which it says will transform the driver experience for the 2 million people who drive for Uber each week. It began about two months ago with the notable addition of tipping to the app. Uber US and Canada manager Rachel Holt told severalpressoutlets that the company's drivers have earned $50 million in tips since that feature was added. In-app tipping became available nationwide in mid-July.
There is good demand for Hadoop integration with Postgres using FDW. Due to same reasonPostgres by BigSQL ships Hadoop FDW package along with other binaries.
let me give a quick step by step instruction (with an example) on how to set it up.
For preparing these steps, I used Hortonworks VM with Docker.
My Hive server is (10.150.11.66) running on port 10000
Step 1. Get Client Library Jars from Hive Server
We should be able to locate the the jar files in the hive server.
$ ls /usr/hdp/*/hadoop/hadoop-common*[0-9].jar
$ ls /usr/hdp/*/hive/lib/hive*jdbc*standalone.jar
Create a directory in the postgres server and copy those two jars into that
$ mkdir hive-client-lib
$ cd hive-client-lib
$ scp -P 2222 firstname.lastname@example.org:/usr/hdp/18.104.22.168-1245/hadoop/hadoop-common-22.214.171.124.5.0.0-1245.jar .
$ scp -P 2222 email@example.com:/usr/hdp/126.96.36.199-1245/hive/lib/hive-jdbc-1.2.1000.2.5.0.0-1245-standalone.jar .
Step 2. Install Java
Hadoop FDW needs Java (JRE) available in Postgres server. In case you don’t have it already, I would recommend installing it with the following pgc command:
$ ./pgc install java8
Get:1 http://10.150.2.165 java8-8u121-linux64
Step 3. Set environment variables and restart PostgreSQL server
I would suggest setting these variables in .profile or .bash_profile or .bashrc:
Restart the PostgreSQL server. This step ensures that the environment variables known to the PostgreSQL server
./pgc restart pg96
Step 4. Create extension and Import the schema / foreign table
Install the extension using the pgc command line utility:
$ ./pgc install hadoop_fdw
Create extension and server definitions:
postgres=# CREATE EXTENSION hadoop_fdw;
postgres=# CREATE SERVER hadoop_server FOREIGN DATA WRAPPER hadoop_fdw OPTIONS (HOST '10.150.11.66', PORT '10000');
postgres=# CREATE USER MAPPING FOR PUBLIC SERVER hadoop_server;
We have the option to import into a Postgres schemaall tables from a Hive database:.
postgres=# import foreign schema “default” from server “hadoop_server” into public;
If we import the entire Hive database, all tables will be visible in the Postgres schema:
List of foreign tables
Schema | Table | Server
public | geolocation | hadoop_server
public | sample_07 | hadoop_server
public | sample_08 | hadoop_server
public | t1 | hadoop_server
public | trucks | hadoop_server
Alternatively, we can create a foreign table in Postgres for a specific table in Hive:
postgres=# CREATE FOREIGN TABLE T1 (
) SERVER hadoop_server OPTIONS (TABLE 'T1');
Now we should be able to query the tables in Hive:
postgres=# SELECT * FROM T1;
id | nm
1 | HADOOP
2 | TESTING USING
3 | HIVE INTERFACE
4 | AND HADOOP FDW
Today is one year since I started working seriously on pg_chameleon. With this commit I changed the project's license to the 2 clause BSD and the project's scope, evolving the project into a MySQL to PostgreSQL replica system.
Initially this change was just a try. I needed to synchronise the data between MySQL and PostgreSQL and at that time the only option I had it was to use the MySQL foreign data wrapper, eventually to copy the data locally every now and then. However, because the previous implementation relied on a MySQL replica this approach approach wasn't really feasible.
If you are curious about the background story and how we scaled the analytics database in Transferwise you can read it here.
I developed pg_chameleon in my spare time. I like to think about it like my little commute project.
The first test on large datasets happened during the amazing days of the pgconf eu 2016. I remember how the process were incredibly slow, taking the unacceptable amount of time. Four days to copy a 600GB database. I found the bottlenecks during the nights between the conference days building a faster implementation.
I also had to cope with the sql dialect conversion. The solution is still in progress.
Currently the project is at the version 1.6 which improves the replay speed and comes with better status view with the replay lag along the read lag.
The upcoming release 1.7 will add an optional threaded mode for the replica, where the read and replay processes will run independently.
This version will also see the support for the type override during the init schema and the ddl replay. This change will make simpler to use pg_chameleon as a migration tool (e.g. conversion of tinyint(1) into a boolean).
However the current replay implementation can result in a broken in case of not compatible data pushed into the data fiels (e.g. insert a value >1 in tinyint(1) will throw a type error on postgres if the data is boolean). I'm working on a solution.
I've also started the development of the version 2 but I've not yet kicked off seriously the coding yet. The reason why is that I'm still learning a lot of things thanks to the feedback I'm getting via github. I will start the version 2 soon and hopefully I will release the first alpha by the beginning of the next year.
However I'm very happy to see pg_chameleon gaining popularity.
If my work even will help just one person to move from MySQL to PostgreSQL, I feel satisfied.
So, happy birthday to pg_chameleon, my little pet project.
La journée de violences de l’opposition s’est soldée par la mort de deux personnes brûlées et un blessé qui a failli être brûlé vif. Dans quelques villes du pays, à l’appel des dirigeants de l’opposition, des voies de communication ont été bloquées et la libre circulation entravée.
Andres Martinez Casares / Reuters
Un homme signalé comme chaviste a été assassiné puis brûlé dans l’État d’Anzoategui, à quelques 240 km de Caracas par un groupe d’opposants, selon les informations de médias locaux.