Monday, July 11, 2016

A rant about OEM lazieness as it pertains to Proprietary Files

Ever wondered why x device slowed down after an upgrade to the newest android version? Or why y doesn't work on a source built ROM?

A BIG part of the issue is OEM's laziness. The bigger part however, is proprietary binaries.

Quite often, proprietary binaries are not updated for feature inclusion, or improvements, and are only updated for the purpose of making the hardware WORK, not WORK WELL. They often go out of their way to actually update the HAL's/RIL's/Proprietary Blobs.


A great example of this is the RIL. The RIL (Radio Interface Layer) is updated EXTREMELY infrequently.

Android's RIL interface is versioned. The current AOSP revision of it is 11. Now, the ril-daemon (Android component that communicates with the RIL) allegedly supports any RIL revision 6=<.

A great example of these components not being updated is in the Nexus S (crespo). It was upgrade to JellyBean, which natively includes RIL version 7. Instead of updating it to get all the perks, bug-fixes, and patches RIL 7 included, they left the prebuilt RIL 6 and just backported code into ril-daemon to support RIL 6. It is no where near optimized for it, and there is often performance degradation due to this. Though this is an old example, it happens on all modern devices (I've seen it on shamu, hlte, etc.) where the OEM's get lazy.


Another common complaint is GPS lag on updated devices. This is due to the same root problem, just more amplified. GPS blobs are commonly static across multiple major android versions. Run a quick MD5 hash on the pre-update version and post-update version, or look at them in IDA (or really, any dissasembler/decompiler). You'll find it rings true.

Name any piece of hardware (including the CPU), and you'll find a case where some OEM decided to back-port old code to avoid doing hard, time consuming work on older devices. For fun some time, grab any OEM's commit log and grep iot for 'hack'. You will be AMAZED how many times they null out functions, or backport archaic paces of code to avoid actually upgrading (and in the process optimizing) their proprietary blobs for new android versions. I'd also image external vendors have some say in this.  Imagine this, you are a Samsung engineer, you have two options;

- Hack together the current proprietary file set to work for the new android version, and ignore optimization.

or

- Contact every one of the external vendors that has devices onboard the device in question, and pay them all to update the blobs, which could also delay the update indefinitely.

Qualcom/Imagtech/Invensense will only update these binaries if they have to/are paid to.


If you read this, you'll pry realize that this problem isn't easily solved, and, to be honest, the way it is currently done is /probably/ the best way to do it in the current world.

Now, you might say, "Can't the community just update the blobs, cause Android is open source?". That question makes me laugh a little to be honest. Android *itself* is open source, as is the RIL. Though, the AOSP RIL will not work on any devices other than AOSP supported devices (excluding a few lucky cases, like some of 2014/15 Moto phone's due to the Nexus 6 being in AOSP and sharing similar hardware). Due to this, most devices don't have an open source RIL. Many people use AOSP-CAF (Qualcom's 'Code Aurora Forums' version of AOSP), which contains a ton of backports, patches, and additional drivers to support a much wider set of devices. This often allows us to update the RIL aftermarket (similar to what was done to bring the OG Moto G up to Marshmallow), but, these don't contain the initial optimizations that OEM's slaved over pre-release on that device, and will always (in all but a few rare cases) perform at least a tad bit crumbier than the OEM blob on its targeted Android version.


Separately, you also have to realize this:

Smartphone's take more physical damage than almost any other high end electronic (yes, even to those of you who take really good care of them). They ride in our pockets being shaken all about for a large portion of their life. They get dropped occasionally, wet, run through a light rain, etc. So, overtime, your hardware takes a real beating. Having opened up many people's older devices for repair, they are FILLED with random crap (excluding certain brands that manage to keep particles out, namely Samsung, which I dislike, as they are the worst offender of the blobs issue described above). They also get dented, etc. Yet we still expect them to last as long as our PC's, which are often treated much better in comparison. Plus, letting the battery (or really, any device in the phone) get above the recommended temperature even once can permanently effect the device's function, or battery's longevity.

And, in all this, I haven't even began to cover NAND/eMMC/UFS aging which slows down I/O read/write speeds, or the aging of other devices, which is amplified in smartphones due to their MASSIVE amount of uptime compared to most devices.

Tl;DR: Just read it, it contains good info.

Friday, April 22, 2016

Guide to Compiling Android on Ubuntu 16.04 (Xenial Xerus) -- OpenJDK7 Installation on 16.04

Another Ubuntu release, another set of instructions to get Android source to compile.



This only applies to the x64 (64 Bit version). It will not compile on the i386 (32 Bit version).

Ubuntu has opted to dump OpenJDK7 in the Xenial Repositories. This just means we will need to add a custom PPA to install JDK7.

Why go with OpenJDK7 and not OpenJDK8?

Well, OpenJDK7 is required to build most popular AOSP 6.0.x ROM Sources as of right now, some examples include: DU, PN, Any Specific AOSP Tag, OmniROM, and Android-x86.

In order to build anything >=4.4.x you will need Oracle Java 6.

The only use case for OpenJDK8 would be those who only build CM (13+), PacROM (or any other CM derivative for that matter), or the 'master' branch of AOSP.


Follow these instructions exactly (don't just throw 'sudo' around either, it is only to be used in specific cases):


1) Remove all traces of Java:

  • sudo apt-get remove openjdk-* icedtea-* icedtea6-*


2) Add Xenial OpenJDK7 PPA & Fetch the new packages index:
  • sudo add-apt-repository ppa:openjdk-r/ppa && sudo apt-get update


3) Install all currently available updates to ensure no packages are broken:
  • sudo apt-get upgrade && sudo apt-get dist-upgrade


6) Install OpenJDK7 and all Android Build dependencies:
  • sudo apt-get install adb fastboot openjdk-7-jdk git ccache automake lzop bison gperf build-essential zip curl zlib1g-dev zlib1g-dev:i386 g++-multilib python-networkx libxml2-utils bzip2 libbz2-dev libbz2-1.0 libghc-bzlib-dev squashfs-tools pngcrush schedtool dpkg-dev liblz4-tool make optipng maven python-mako python3-mako python python3 syslinux-utils google-android-build-tools-installer

# To install OpenJDK8, replace 'openjdk-7-jdk' ==> 'openjdk-8-jdk'
# To install Oracle Java 6, replace 'openjdk-7-jdk' ==> 'oracle-java6-installer'
# You can install all three Java versions, you will just need to select the default using the 'update-alternatives' command below



7) Remove all unnecessary packages:
  • sudo apt-get autoremove


8) Make a user accessible folder, and add it to path:
  • mkdir ~/bin && echo "export PATH=~/bin:$PATH" >> ~/.bashrc


9) Enable CCACHE to speed up builds:
  • echo "export USE_CCACHE=1" >> ~/.bashrc


10) Restart Bash
  • source ~/.bashrc


11) Configure Git (Replace with your Name & Email Address):

  • git config --global user.name "John Doe"
  • git config --global user.email johndoe@example.com


12) Download Repo, make it executable, and put it in our local $PATH folder:

  • curl http://commondatastorage.googleapis.com/git-repo-downloads/repo > ~/bin/repo && chmod a+x ~/bin/repo


13) Make a folder for your ROM source of choice (in this case 'android'), and initialize the repo:
  • mkdir ~/android
  • cd ~/android
  • repo init -u UrlOfManifestofROM.git -b BranchName


14) Sync the Repo:
  • repo sync -j4 --force-sync


15) Setup your build enviroment:
  • . build/envsetup.sh


16) Choose here:

# If you want to fetch device dependencies, and your ROM officially supports the target device -- goto step 16) a)

# If you want to build using local device tree, repo, and kernel directories -- goto step 16) b)


16) a) Breakfast
  • breakfast RomTag_DeviceCodeName-BuildType

# I'll use DirtyUnicorns on as an example

#'breakfast du_angler-userdebug' (Builds DU for Nexus 6P with userdebug, unless you know what you are doing, use userdebug)

# If you don't know your device codename, Google it, or just run 'breakfast', and look for it in the menu


16) b) Lunch
  • lunch RomTag_DeviceCodeName-BuildType

# I'll use DirtyUnicorns on as an example

#'lunch du_angler-userdebug' (Builds DU for Nexus 6P with userdebug, unless you know what you are doing, use userdebug)

# If you don't know your device codename, Google it, or just run 'lunch', and look for it in the menu



17) Build the ROM:
  • mka -j8 bacon

# mka is a refined make.

# -j8 is a variable number, if you have an Intel processor, take the number of cores and multiply by two, so, on my Quad-Core i5, 4x2=8. As noted by +Drew Walton, the 'mka' command auto analyses cpu availabillity, so the '-jx' arguement is not necessary unless you are building on a device where you want limited resources to be used (I have a server, where I am only allowed to use 4 cores to build, so 2 x 4 due to hyperthreading).

# bacon is most custom ROM's target for a flash-able .zip archive format, if you get an error like "No Target to Make: bacon", try 'make -j8 otapackage'.


18) Collect the ROM:
  • adb push ~/android/out/target/product/ROMNAME-DATE.zip /sdcard/

19) Flash ROM & Enjoy!



NOTES:
- Some ROMs don't sync in vendor repos by default. You'll need to do this by fetching your OEM's vendor from https://github.com/TheMuppets and cloning it into ~/PathToROMSource/vendor/OEMName. You can also do this via local manifest. Look at this one as an example. but I won't be explaining it here: https://github.com/adt1-dev/local_manifests/tree/cm-13.0.

- Getting random Java or Host Binary (build output shows it building in the 'out/host' directory) related errors? Try a lower -j#, or, just exclude that part (it will default to -j1, which will take considerably longer).

- To change your default Java version, use these two commands (Make sure to use them both! Mix and matching Java versions will not go well!):

sudo update-alternatives --config java

# If building LP (Lollipop) - MM (Marshmallow): Select the option for 'java-7-openjdk-amd64'
# If building DN (Donut) - KK (KitKat): Select the option for 'java-6-oracle-amd64'
# If building CM (13+), Pac (Or any other CM derivative), or the 'master' branch of AOSP: Select the option for 'java-8-openjdk-amd64'

sudo update-alternatives --config javac

# If building LP (Lollipop) - MM (Marshmallow): Select the option for 'java-7-openjdk-amd64'
# If building DN (Donut) - KK (KitKat): Select the option for 'java-6-oracle-amd64'
# If building CM (13+), Pac (Or any other CM derivative), or the 'master' branch of AOSP: Select the option for 'java-8-openjdk-amd64'



NOTE: Some AOSP ROM Source trees have been updated to build with OpenJDK8, so, feel free to use it when you can, though, it isn't mandatory.

Getting Random HOST ART Library Errors?

A common error going around right now is the generic 'libart.so' failure to build due to a linker error. This is due to a recent update in the BinUtils package in the Ubuntu and Debian repositories that AOSP hasn't accounted for yet (CM already merged the fix). To fix this, you need to cherry-pick two commits:
  • cd ~/PathToROMSource
  • cd art/build/
  •  git remote add cm https://github.com/CyanogenMod/android_art.git
  • git fetch cm
  • git cherry-pick e5c6b049f3c716be60bc82d79c44c451f49b4ad5
  • git cherry-pick 0299788b7a974841aa3324a573fbf04f49a4f23c
  • # Thats it! Have fun building your ROM of choice from source!