Implementing Android incremental loader

 Программирование  android

There is an outstanding article on implementing Android loaders. However it covers classic pattern when data is being reloaded each time any part of it is modified. In my case I had to manipulate huge files, some of them taking minutes to load, so I could not afford reloading them on each data set change.

That's why I have extended logic of my Loader to support incremental loading.

First, we define structure that will hold data state:

private final Set<String> mFiles = new HashSet<>();

We hold loaded data as usual:

private List<DataSource> mData;

When Loader starts it delivers already available data and initiates data change observer. If any data is added or removed data state holder is updated.

protected void onStartLoading() {
    if (mData != null) {
        // Deliver any previously loaded data immediately, note that
        // deliverResult() gets newly added data instead of full data
        // in our case
        deliverResult(new ArrayList<DataSource>());

    // Begin monitoring the underlying data source
    if (mObserver == null) {
        final File dir = getContext().getExternalFilesDir("data");
        if (dir == null)

        mObserver = new FileObserver(dir.getAbsolutePath(),
                        FileObserver.CLOSE_WRITE |
                        FileObserver.MOVED_FROM |
                        FileObserver.MOVED_TO |
                        FileObserver.DELETE) {
            public void onEvent(int event, String path) {
                if (event == 0x8000) // Undocumented, sent on stop watching
                if (path == null) // Undocumented, unexplainable
                path = dir.getAbsolutePath() + File.separator + path;
                synchronized (mFiles) {
                    boolean loadedSource = false;
                    for (Iterator<DataSource> i = mData.iterator(); i.hasNext(); ) {
                        DataSource source =;
                        if (source.path.equals(path)) {
                            if (source.isLoaded())
                                loadedSource = true;
                    if (!loadedSource) {
                        // Indicate state change by simply removing file
                        // reference from state holder

    if (takeContentChanged() || mData == null) {

Background loading starts, but in our case it should return only updated delta, not the full data set. We check data state and skip previously loaded data.

public List<DataSource> loadInBackground() {
    Context ctx = getContext();
    File dataDir = ctx.getExternalFilesDir("data");
    if (dataDir == null)
        return null;
    File[] files = dataDir.listFiles();
    if (files == null)
        return null;

    List<DataSource> data = new ArrayList<>();

    for (File file : files) {
        if (isLoadInBackgroundCanceled())
            return null;
        synchronized (mFiles) {
            // Skip already loaded data
            if (mFiles.contains(path))
        try {
            FileInputStream inputStream = new FileInputStream(file));
            DataSource source = loadData(inputStream;
            source.path = file.getAbsolutePath();
        } catch (Exception e) {
    return data;

When result is ready we join existing data with newly loaded data.

public void deliverResult(List<DataSource> data) {
    if (isReset()) {

    synchronized (mFiles) {
        if (mData == null) {
            mData = data;
        } else {
            // Somewhere under the hood it looks if the data has changed
            // by comparing object instances, so we need a new object
            ArrayList<DataSource> newData = new ArrayList<>(mData.size());
            mData = newData;

        for (DataSource source : data) {

    if (isStarted())

Data is delivered to consumer in a full set but all object instances are preserved thus not breaking the data manipulation logic of data consumer.

Here arrives a good question: what happens when data consumer updates or deletes some data? This means that corresponding file will be modified or removed and observer will detect that. This case is already handled by the code. May be you have noticed a call to source.isLoaded() - this is the flag of internal data state. When data is loaded and consumer then modifies it, flag is set and Loader just skips the corresponding file. If data is wished to be removed, corresponding file is deleted and that flag is unset. In that case Loader removes data associated with this file.